2024-04-22T00:53:51.929 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T00:53:51.929 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/orig.config.yaml 2024-04-22T00:53:51.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T00:53:52.092 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667718' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 676 sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b sleep_before_teardown: 0 subset: 46/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLGQtVNeI9+UFHWNWwuuOzFKtEOrBClAunKnuf1Vzp2MfNCTzOxPjd3R4nSfQ8Fdr6IBbJ9gc1Xua4Wf20XP4A= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBML8zKbUiePm5I3bXabHl/Ln0kFY5/Wkm342BA/XKIw2sJopiUv1XJC7BH+tFFiObI2ETHO41wdFLkKPk8HfnZA= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+thXPC3JWrixBoXHYchEzRgYrwQRZxcAUp2wMPV1CPZVv5Sd4iYsE7u5CzoUMwCGkLH/6UBOXGnsK6cITHyPs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T00:53:52.092 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa; will attempt to use it 2024-04-22T00:53:52.093 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks 2024-04-22T00:53:52.093 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T00:53:52.095 INFO:teuthology.task.internal:Checking packages... 2024-04-22T00:53:52.117 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '820a2f8ced66152ed3d765c31444b8076ea28c6b' 2024-04-22T00:53:52.117 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T00:53:52.117 INFO:teuthology.packaging:ref: None 2024-04-22T00:53:52.117 INFO:teuthology.packaging:tag: None 2024-04-22T00:53:52.117 INFO:teuthology.packaging:branch: squid 2024-04-22T00:53:52.118 INFO:teuthology.packaging:sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T00:53:52.118 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-22T00:53:52.442 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2407.g430e09df 2024-04-22T00:53:52.443 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T00:53:52.454 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T00:53:52.454 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T00:53:52.468 INFO:teuthology.task.internal:Saving configuration 2024-04-22T00:53:52.488 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T00:53:52.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T00:53:52.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 00:47:44.156943', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLGQtVNeI9+UFHWNWwuuOzFKtEOrBClAunKnuf1Vzp2MfNCTzOxPjd3R4nSfQ8Fdr6IBbJ9gc1Xua4Wf20XP4A='} 2024-04-22T00:53:52.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 00:47:44.161277', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBML8zKbUiePm5I3bXabHl/Ln0kFY5/Wkm342BA/XKIw2sJopiUv1XJC7BH+tFFiObI2ETHO41wdFLkKPk8HfnZA='} 2024-04-22T00:53:52.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 00:47:44.158930', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+thXPC3JWrixBoXHYchEzRgYrwQRZxcAUp2wMPV1CPZVv5Sd4iYsE7u5CzoUMwCGkLH/6UBOXGnsK6cITHyPs='} 2024-04-22T00:53:52.575 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T00:53:52.587 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-22T00:53:52.587 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-22T00:53:52.587 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-22T00:53:52.587 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T00:53:52.700 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa560803160>, signals=[15]) 2024-04-22T00:53:52.701 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T00:53:52.711 INFO:teuthology.task.internal:Opening connections... 2024-04-22T00:53:52.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:53:52.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T00:53:52.788 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-04-22T00:53:52.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T00:53:52.861 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-22T00:53:52.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T00:53:52.934 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T00:53:52.947 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-22T00:53:52.967 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-22T00:53:52.967 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-22T00:53:53.025 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-22T00:53:53.025 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-22T00:53:53.025 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-22T00:53:53.025 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-22T00:53:53.025 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T00:53:53.026 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T00:53:53.027 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-22T00:53:53.055 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-04-22T00:53:53.073 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-04-22T00:53:53.073 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-04-22T00:53:53.129 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-04-22T00:53:53.129 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-04-22T00:53:53.130 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T00:53:53.131 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T00:53:53.131 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T00:53:53.131 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-04-22T00:53:53.152 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-22T00:53:53.170 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-22T00:53:53.170 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-22T00:53:53.227 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T00:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T00:53:53.228 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-22T00:53:53.252 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T00:53:53.273 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T00:53:53.281 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T00:53:53.281 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-22T00:53:53.284 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-04-22T00:53:53.287 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-22T00:53:53.304 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T00:53:53.312 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T00:53:53.312 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-22T00:53:53.342 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-04-22T00:53:53.344 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-22T00:53:53.585 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T00:53:53.666 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T00:53:53.686 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-22T00:53:53.687 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-22T00:53:53.687 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-22T00:53:53.687 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T00:53:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:53:53.704 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-04-22T00:53:53.761 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-437.el9.x86_64 2024-04-22T00:53:53.762 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-437.el9.x86_64 2024-04-22T00:53:53.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-04-22T00:53:56.254 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T00:54:00.821 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-04-22T00:54:12.571 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 781 kB/s | 6.2 MB 00:08 2024-04-22T00:54:14.981 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-22T00:54:16.032 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-22T00:54:21.182 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-22T00:54:23.172 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T00:54:23.172 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T00:54:23.259 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T00:54:23.261 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-22T00:54:23.261 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T00:54:23.361 DEBUG:teuthology.orchestra.run.smithi038:> echo no | sudo yum reinstall kernel || true 2024-04-22T00:54:23.881 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 12:54:21 AM UTC. 2024-04-22T00:54:24.004 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T00:54:24.015 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T00:54:24.015 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-22T00:54:24.015 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T00:54:24.015 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-22T00:54:24.016 INFO:teuthology.orchestra.run.smithi038.stderr:Operation aborted. 2024-04-22T00:54:24.074 INFO:teuthology.orchestra.run.smithi038.stdout:Is this ok [y/N]: 2024-04-22T00:54:24.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum reinstall -y kernel || true 2024-04-22T00:54:24.589 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 12:54:21 AM UTC. 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T00:54:24.711 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:54:24.712 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T00:54:24.712 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-22T00:54:24.712 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:54:24.712 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-22T00:54:24.712 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-22T00:54:24.712 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T00:54:24.993 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-22T00:54:24.993 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T00:54:24.993 INFO:teuthology.orchestra.run.smithi038.stdout:Total 908 kB/s | 255 kB 00:00 2024-04-22T00:54:25.010 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T00:54:25.201 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T00:54:25.201 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T00:54:25.206 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T00:54:25.206 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T00:54:25.338 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T00:54:25.419 INFO:teuthology.orchestra.run.smithi038.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T00:54:25.419 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T00:54:26.099 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T00:54:26.100 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T00:54:26.777 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T00:54:26.778 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:54:26.778 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalled: 2024-04-22T00:54:26.778 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T00:54:26.778 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:54:26.778 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T00:54:26.956 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T00:54:27.004 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T00:54:27.004 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T00:54:27.004 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T00:54:27.005 DEBUG:teuthology.parallel:result is None 2024-04-22T00:54:27.005 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T00:54:27.016 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T00:54:27.016 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T00:54:27.022 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T00:54:27.025 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T00:54:27.043 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T00:54:27.050 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T00:54:27.057 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T00:54:27.057 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T00:54:27.073 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T00:54:27.082 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T00:54:27.114 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T00:54:27.120 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T00:54:27.121 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T00:54:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:54:27.162 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T00:54:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:54:27.179 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T00:54:27.181 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T00:54:27.205 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T00:54:27.222 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T00:54:27.235 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T00:54:27.256 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T00:54:27.258 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T00:54:27.272 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T00:54:27.272 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T00:54:27.274 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T00:54:27.282 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T00:54:27.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T00:54:27.284 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T00:54:27.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T00:54:27.344 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T00:54:27.352 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T00:54:27.352 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T00:54:27.354 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T00:54:27.382 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T00:54:27.400 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T00:54:27.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T00:54:27.507 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T00:54:27.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T00:54:27.629 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:54:27.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T00:54:27.693 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T00:54:27.738 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T00:54:27.789 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T00:54:27.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T00:54:27.911 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T00:54:27.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T00:54:27.976 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T00:54:28.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T00:54:28.079 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T00:54:28.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T00:54:28.205 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T00:54:28.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T00:54:28.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-22T00:54:28.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-04-22T00:54:28.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-22T00:54:28.309 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T00:54:28.310 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T00:54:28.342 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T00:54:28.798 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T00:54:28.822 INFO:teuthology.task.internal:Starting timer... 2024-04-22T00:54:28.822 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T00:54:28.832 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T00:54:28.841 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T00:54:28.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-22T00:54:28.924 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-22T00:54:28.927 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-04-22T00:54:28.970 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-04-22T00:54:28.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-22T00:54:29.037 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-22T00:54:29.039 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T00:54:29.039 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-22T00:54:29.057 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-22T00:54:29.057 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-04-22T00:54:29.080 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-04-22T00:54:29.080 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-22T00:54:29.118 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-22T00:54:29.118 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-22T00:54:29.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T00:54:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:54:29.144 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T00:54:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:54:29.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T00:54:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:54:29.199 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T00:54:29.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-22T00:54:29.223 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-04-22T00:54:29.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-22T00:54:29.274 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T00:54:29.292 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T00:54:29.309 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T00:54:29.311 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi111.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T00:58:38.552 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-22T00:58:38.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-22T00:58:38.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T00:58:38.631 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-22T00:58:38.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-22T00:58:38.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-22T00:58:38.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T00:58:38.784 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-22T00:58:38.870 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-22T00:58:38.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-22T00:58:38.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T00:58:38.945 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-22T00:58:39.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-22T00:58:39.017 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T00:58:39.028 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T00:58:39.028 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T00:58:39.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T00:58:39.031 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T00:58:39.031 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T00:58:39.034 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T00:58:39.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T00:58:39.060 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T00:58:39.062 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T00:58:39.075 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T00:58:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T00:58:39.084 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T00:58:39.098 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T00:58:39.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-22T00:58:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-04-22T00:58:39.112 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-22T00:58:39.112 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-04-22T00:58:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-22T00:58:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T00:58:39.125 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T00:58:39.130 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-22T00:58:39.138 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T00:58:39.139 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T00:58:39.143 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T00:58:39.156 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T00:58:39.196 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-22T00:58:39.197 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-22T00:58:39.199 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T00:58:39.199 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-22T00:58:39.199 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.200 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.200 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.200 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.200 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T00:58:39.201 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-22T00:58:39.201 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.201 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.201 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.201 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.215 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-22T00:58:39.218 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T00:58:39.219 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-22T00:58:39.219 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.219 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.219 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.219 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:58:39.220 INFO:teuthology.run_tasks:Running task install... 2024-04-22T00:58:39.228 DEBUG:teuthology.task.install:project ceph 2024-04-22T00:58:39.228 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}} 2024-04-22T00:58:39.228 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'} 2024-04-22T00:58:39.228 INFO:teuthology.task.install:Using flavor: default 2024-04-22T00:58:39.234 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T00:58:39.234 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-22T00:58:39.234 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-22T00:58:39.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T00:58:39.237 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-22T00:58:39.237 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T00:58:39.238 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-22T00:58:39.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T00:58:39.377 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-22T00:58:39.377 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-22T00:58:39.387 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-22T00:58:39.387 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-22T00:58:39.461 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-22T00:58:39.461 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-22T00:58:39.495 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T00:58:39.495 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T00:58:39.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T00:58:39.499 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T00:58:39.499 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T00:58:39.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T00:58:39.520 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T00:58:39.521 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T00:58:39.525 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T00:58:39.525 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T00:58:39.571 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T00:58:39.571 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:58:39.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T00:58:39.585 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T00:58:39.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T00:58:39.598 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T00:58:39.599 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T00:58:39.658 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T00:58:39.662 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T00:58:39.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T00:58:39.721 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-04-22T00:58:39.722 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-04-22T00:58:39.727 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-22T00:58:39.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-22T00:58:39.739 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T00:58:39.804 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-22T00:58:39.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-22T00:58:39.947 INFO:teuthology.orchestra.run.smithi111.stdout:48 files removed 2024-04-22T00:58:39.953 INFO:teuthology.orchestra.run.smithi203.stdout:48 files removed 2024-04-22T00:58:39.974 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T00:58:39.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T00:58:40.039 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-22T00:58:40.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T00:58:40.654 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 203 kB/s | 83 kB 00:00 2024-04-22T00:58:40.699 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 231 kB/s | 83 kB 00:00 2024-04-22T00:58:40.703 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 177 kB/s | 83 kB 00:00 2024-04-22T00:58:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-04-22T00:58:41.045 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-04-22T00:58:41.091 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-04-22T00:58:41.395 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-04-22T00:58:41.439 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 4.7 kB/s | 1.8 kB 00:00 2024-04-22T00:58:41.475 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 5.3 kB/s | 1.8 kB 00:00 2024-04-22T00:58:42.062 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-22T00:58:42.151 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-22T00:58:44.030 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-22T00:58:48.123 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-22T00:58:50.406 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-22T00:58:50.622 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 901 kB/s | 8.0 MB 00:09 2024-04-22T00:58:51.770 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-22T00:58:52.901 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-22T00:58:56.783 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-22T00:58:57.107 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.2 MB 00:00 2024-04-22T00:58:58.784 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-22T00:58:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T00:58:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T00:58:58.928 INFO:teuthology.orchestra.run.smithi111.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T00:58:58.929 INFO:teuthology.orchestra.run.smithi111.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T00:58:58.930 INFO:teuthology.orchestra.run.smithi111.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T00:58:58.931 INFO:teuthology.orchestra.run.smithi111.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T00:58:58.932 INFO:teuthology.orchestra.run.smithi111.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T00:58:58.935 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T00:58:58.935 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T00:58:58.995 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-22T00:58:59.047 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-22T00:58:59.048 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-22T00:58:59.049 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T00:58:59.050 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T00:58:59.051 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T00:58:59.052 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-22T00:58:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T00:58:59.054 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T00:58:59.055 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T00:58:59.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T00:58:59.057 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Install 148 Packages 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 216 M 2024-04-22T00:58:59.058 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-22T00:58:59.743 INFO:teuthology.orchestra.run.smithi111.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-22T00:58:59.768 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-22T00:59:00.010 INFO:teuthology.orchestra.run.smithi111.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 3.3 MB/s | 889 kB 00:00 2024-04-22T00:59:00.068 INFO:teuthology.orchestra.run.smithi111.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 2.4 MB/s | 145 kB 00:00 2024-04-22T00:59:00.385 INFO:teuthology.orchestra.run.smithi111.stdout:(4/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 7.5 MB/s | 2.4 MB 00:00 2024-04-22T00:59:00.494 INFO:teuthology.orchestra.run.smithi111.stdout:(5/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 5.8 MB/s | 5.4 MB 00:00 2024-04-22T00:59:00.610 INFO:teuthology.orchestra.run.smithi111.stdout:(6/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 6.6 MB/s | 1.5 MB 00:00 2024-04-22T00:59:01.027 INFO:teuthology.orchestra.run.smithi111.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 8.8 MB/s | 4.7 MB 00:00 2024-04-22T00:59:02.302 INFO:teuthology.orchestra.run.smithi111.stdout:(8/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 8.0 MB/s | 10 MB 00:01 2024-04-22T00:59:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 336 kB/s | 25 kB 00:00 2024-04-22T00:59:02.786 INFO:teuthology.orchestra.run.smithi111.stdout:(10/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x8 7.7 MB/s | 17 MB 00:02 2024-04-22T00:59:02.886 INFO:teuthology.orchestra.run.smithi111.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 2.2 MB/s | 222 kB 00:00 2024-04-22T00:59:02.978 INFO:teuthology.orchestra.run.smithi111.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 295 kB/s | 27 kB 00:00 2024-04-22T00:59:03.070 INFO:teuthology.orchestra.run.smithi111.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 346 kB/s | 32 kB 00:00 2024-04-22T00:59:03.379 INFO:teuthology.orchestra.run.smithi111.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 2.4 MB/s | 745 kB 00:00 2024-04-22T00:59:03.471 INFO:teuthology.orchestra.run.smithi111.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.7 MB/s | 164 kB 00:00 2024-04-22T00:59:03.563 INFO:teuthology.orchestra.run.smithi111.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.4 MB/s | 127 kB 00:00 2024-04-22T00:59:03.796 INFO:teuthology.orchestra.run.smithi111.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 2.1 MB/s | 501 kB 00:00 2024-04-22T00:59:04.927 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-22T00:59:05.208 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 887 kB/s | 19 MB 00:21 2024-04-22T00:59:05.313 INFO:teuthology.orchestra.run.smithi111.stdout:(18/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 3.5 MB/s | 5.3 MB 00:01 2024-04-22T00:59:05.397 INFO:teuthology.orchestra.run.smithi111.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 543 kB/s | 45 kB 00:00 2024-04-22T00:59:05.480 INFO:teuthology.orchestra.run.smithi111.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.6 MB/s | 133 kB 00:00 2024-04-22T00:59:05.572 INFO:teuthology.orchestra.run.smithi111.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 1.7 MB/s | 161 kB 00:00 2024-04-22T00:59:05.731 INFO:teuthology.orchestra.run.smithi111.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 2.0 MB/s | 321 kB 00:00 2024-04-22T00:59:05.840 INFO:teuthology.orchestra.run.smithi111.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 2.7 MB/s | 298 kB 00:00 2024-04-22T00:59:05.932 INFO:teuthology.orchestra.run.smithi111.stdout:(24/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.1 MB/s | 99 kB 00:00 2024-04-22T00:59:06.182 INFO:teuthology.orchestra.run.smithi111.stdout:(25/150): ceph-common-19.0.0-2405.g820a2f8c.el9 3.2 MB/s | 21 MB 00:06 2024-04-22T00:59:06.207 INFO:teuthology.orchestra.run.smithi111.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 311 kB/s | 86 kB 00:00 2024-04-22T00:59:06.308 INFO:teuthology.orchestra.run.smithi111.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-22T00:59:06.408 INFO:teuthology.orchestra.run.smithi111.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 318 kB/s | 32 kB 00:00 2024-04-22T00:59:06.500 INFO:teuthology.orchestra.run.smithi111.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.5 MB/s | 141 kB 00:00 2024-04-22T00:59:07.070 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T00:59:07.070 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T00:59:07.081 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T00:59:07.082 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T00:59:07.083 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T00:59:07.083 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T00:59:07.085 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T00:59:07.087 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T00:59:07.088 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T00:59:07.147 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-22T00:59:07.170 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-22T00:59:07.171 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T00:59:07.172 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-22T00:59:07.173 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-22T00:59:07.174 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-22T00:59:07.175 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T00:59:07.176 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T00:59:07.177 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T00:59:07.178 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T00:59:07.179 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T00:59:07.180 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T00:59:07.181 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout:Install 148 Packages 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-04-22T00:59:07.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T00:59:07.183 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 216 M 2024-04-22T00:59:07.183 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-22T00:59:07.325 INFO:teuthology.orchestra.run.smithi111.stdout:(30/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 4.3 MB/s | 3.5 MB 00:00 2024-04-22T00:59:07.751 INFO:teuthology.orchestra.run.smithi111.stdout:(31/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 2.0 MB/s | 3.1 MB 00:01 2024-04-22T00:59:08.009 INFO:teuthology.orchestra.run.smithi111.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 964 kB/s | 249 kB 00:00 2024-04-22T00:59:08.118 INFO:teuthology.orchestra.run.smithi111.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 456 kB/s | 49 kB 00:00 2024-04-22T00:59:08.210 INFO:teuthology.orchestra.run.smithi111.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 167 kB/s | 15 kB 00:00 2024-04-22T00:59:08.418 INFO:teuthology.orchestra.run.smithi111.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 1.2 MB/s | 264 kB 00:00 2024-04-22T00:59:08.653 INFO:teuthology.orchestra.run.smithi203.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-22T00:59:08.735 INFO:teuthology.orchestra.run.smithi111.stdout:(36/150): ceph-mgr-diskprediction-local-19.0.0- 5.2 MB/s | 7.4 MB 00:01 2024-04-22T00:59:08.811 INFO:teuthology.orchestra.run.smithi111.stdout:(37/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 244 kB/s | 18 kB 00:00 2024-04-22T00:59:08.869 INFO:teuthology.orchestra.run.smithi111.stdout:(38/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 1.7 MB/s | 762 kB 00:00 2024-04-22T00:59:09.019 INFO:teuthology.orchestra.run.smithi111.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 385 kB/s | 80 kB 00:00 2024-04-22T00:59:09.111 INFO:teuthology.orchestra.run.smithi111.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-22T00:59:09.145 INFO:teuthology.orchestra.run.smithi111.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-04-22T00:59:09.170 INFO:teuthology.orchestra.run.smithi111.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-22T00:59:09.229 INFO:teuthology.orchestra.run.smithi111.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-22T00:59:09.271 INFO:teuthology.orchestra.run.smithi111.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.2 MB 00:00 2024-04-22T00:59:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 135 kB 00:00 2024-04-22T00:59:09.330 INFO:teuthology.orchestra.run.smithi111.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-22T00:59:09.364 INFO:teuthology.orchestra.run.smithi111.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-22T00:59:09.478 INFO:teuthology.orchestra.run.smithi203.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.1 MB/s | 889 kB 00:00 2024-04-22T00:59:09.597 INFO:teuthology.orchestra.run.smithi111.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 689 kB 00:00 2024-04-22T00:59:09.664 INFO:teuthology.orchestra.run.smithi111.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-04-22T00:59:09.670 INFO:teuthology.orchestra.run.smithi203.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 755 kB/s | 145 kB 00:00 2024-04-22T00:59:09.689 INFO:teuthology.orchestra.run.smithi111.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-22T00:59:09.731 INFO:teuthology.orchestra.run.smithi111.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-04-22T00:59:09.782 INFO:teuthology.orchestra.run.smithi111.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-22T00:59:10.051 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.2 MB 00:01 2024-04-22T00:59:10.199 INFO:teuthology.orchestra.run.smithi111.stdout:(53/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 6.2 MB/s | 48 MB 00:07 2024-04-22T00:59:10.224 INFO:teuthology.orchestra.run.smithi111.stdout:(54/150): flexiblas-3.0.4-8.el9.x86_64.rpm 66 kB/s | 33 kB 00:00 2024-04-22T00:59:10.282 INFO:teuthology.orchestra.run.smithi111.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 192 kB/s | 11 kB 00:00 2024-04-22T00:59:10.396 INFO:teuthology.orchestra.run.smithi203.stdout:(4/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 2.4 MB/s | 5.4 MB 00:02 2024-04-22T00:59:10.401 INFO:teuthology.orchestra.run.smithi111.stdout:(56/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.8 MB/s | 3.0 MB 00:00 2024-04-22T00:59:10.425 INFO:teuthology.orchestra.run.smithi111.stdout:(57/150): flexiblas-openblas-openmp-3.0.4-8.el9 91 kB/s | 18 kB 00:00 2024-04-22T00:59:10.450 INFO:teuthology.orchestra.run.smithi111.stdout:(58/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-04-22T00:59:10.475 INFO:teuthology.orchestra.run.smithi111.stdout:(59/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 45 kB 00:00 2024-04-22T00:59:10.517 INFO:teuthology.orchestra.run.smithi111.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-04-22T00:59:10.559 INFO:teuthology.orchestra.run.smithi111.stdout:(61/150): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-22T00:59:10.584 INFO:teuthology.orchestra.run.smithi111.stdout:(62/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-04-22T00:59:10.610 INFO:teuthology.orchestra.run.smithi111.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-04-22T00:59:10.635 INFO:teuthology.orchestra.run.smithi111.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-22T00:59:10.660 INFO:teuthology.orchestra.run.smithi111.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-22T00:59:10.685 INFO:teuthology.orchestra.run.smithi111.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-22T00:59:10.796 INFO:teuthology.orchestra.run.smithi203.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 3.7 MB/s | 1.5 MB 00:00 2024-04-22T00:59:10.802 INFO:teuthology.orchestra.run.smithi111.stdout:(67/150): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-04-22T00:59:10.871 INFO:teuthology.orchestra.run.smithi203.stdout:(6/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 2.0 MB/s | 2.4 MB 00:01 2024-04-22T00:59:10.886 INFO:teuthology.orchestra.run.smithi111.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-04-22T00:59:10.919 INFO:teuthology.orchestra.run.smithi111.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-04-22T00:59:10.970 INFO:teuthology.orchestra.run.smithi111.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-22T00:59:11.062 INFO:teuthology.orchestra.run.smithi111.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-22T00:59:11.162 INFO:teuthology.orchestra.run.smithi111.stdout:(72/150): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-04-22T00:59:11.187 INFO:teuthology.orchestra.run.smithi111.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 699 kB/s | 87 kB 00:00 2024-04-22T00:59:11.430 INFO:teuthology.orchestra.run.smithi111.stdout:(74/150): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-04-22T00:59:11.463 INFO:teuthology.orchestra.run.smithi111.stdout:(75/150): postgresql-13.14-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-04-22T00:59:11.521 INFO:teuthology.orchestra.run.smithi111.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-04-22T00:59:11.547 INFO:teuthology.orchestra.run.smithi111.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 306 kB/s | 26 kB 00:00 2024-04-22T00:59:11.614 INFO:teuthology.orchestra.run.smithi111.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 19 kB 00:00 2024-04-22T00:59:11.635 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-22T00:59:11.639 INFO:teuthology.orchestra.run.smithi111.stdout:(79/150): python3-devel-3.9.18-3.el9.x86_64.rpm 540 kB/s | 244 kB 00:00 2024-04-22T00:59:11.664 INFO:teuthology.orchestra.run.smithi111.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-22T00:59:11.690 INFO:teuthology.orchestra.run.smithi111.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 459 kB/s | 35 kB 00:00 2024-04-22T00:59:11.740 INFO:teuthology.orchestra.run.smithi111.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-22T00:59:11.782 INFO:teuthology.orchestra.run.smithi111.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-22T00:59:11.824 INFO:teuthology.orchestra.run.smithi111.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 77 kB 00:00 2024-04-22T00:59:11.899 INFO:teuthology.orchestra.run.smithi111.stdout:(85/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-04-22T00:59:11.924 INFO:teuthology.orchestra.run.smithi111.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 297 kB/s | 42 kB 00:00 2024-04-22T00:59:11.949 INFO:teuthology.orchestra.run.smithi111.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-22T00:59:11.975 INFO:teuthology.orchestra.run.smithi111.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-22T00:59:12.000 INFO:teuthology.orchestra.run.smithi111.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-22T00:59:12.025 INFO:teuthology.orchestra.run.smithi111.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-04-22T00:59:12.030 INFO:teuthology.orchestra.run.smithi203.stdout:(7/150): ceph-common-19.0.0-2405.g820a2f8c.el9. 5.4 MB/s | 21 MB 00:03 2024-04-22T00:59:12.050 INFO:teuthology.orchestra.run.smithi111.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 713 kB/s | 54 kB 00:00 2024-04-22T00:59:12.109 INFO:teuthology.orchestra.run.smithi111.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 499 kB/s | 42 kB 00:00 2024-04-22T00:59:12.143 INFO:teuthology.orchestra.run.smithi111.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 480 kB/s | 44 kB 00:00 2024-04-22T00:59:12.226 INFO:teuthology.orchestra.run.smithi111.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-04-22T00:59:12.251 INFO:teuthology.orchestra.run.smithi111.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 330 kB/s | 36 kB 00:00 2024-04-22T00:59:12.310 INFO:teuthology.orchestra.run.smithi111.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-04-22T00:59:12.486 INFO:teuthology.orchestra.run.smithi111.stdout:(97/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-04-22T00:59:12.511 INFO:teuthology.orchestra.run.smithi111.stdout:(98/150): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-04-22T00:59:12.536 INFO:teuthology.orchestra.run.smithi111.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 256 kB/s | 58 kB 00:00 2024-04-22T00:59:12.561 INFO:teuthology.orchestra.run.smithi111.stdout:(100/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-22T00:59:12.586 INFO:teuthology.orchestra.run.smithi111.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 845 kB/s | 64 kB 00:00 2024-04-22T00:59:12.612 INFO:teuthology.orchestra.run.smithi111.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 594 kB/s | 45 kB 00:00 2024-04-22T00:59:12.938 INFO:teuthology.orchestra.run.smithi203.stdout:(8/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 11 MB/s | 10 MB 00:00 2024-04-22T00:59:13.005 INFO:teuthology.orchestra.run.smithi203.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 379 kB/s | 25 kB 00:00 2024-04-22T00:59:13.162 INFO:teuthology.orchestra.run.smithi111.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-04-22T00:59:13.221 INFO:teuthology.orchestra.run.smithi111.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 900 kB/s | 548 kB 00:00 2024-04-22T00:59:13.330 INFO:teuthology.orchestra.run.smithi111.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-04-22T00:59:13.722 INFO:teuthology.orchestra.run.smithi111.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-22T00:59:13.747 INFO:teuthology.orchestra.run.smithi111.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 529 kB/s | 309 kB 00:00 2024-04-22T00:59:13.822 INFO:teuthology.orchestra.run.smithi111.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-22T00:59:13.847 INFO:teuthology.orchestra.run.smithi111.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-04-22T00:59:13.897 INFO:teuthology.orchestra.run.smithi203.stdout:(10/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x8 1.5 MB/s | 4.7 MB 00:03 2024-04-22T00:59:13.923 INFO:teuthology.orchestra.run.smithi111.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-04-22T00:59:13.973 INFO:teuthology.orchestra.run.smithi111.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-22T00:59:14.025 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 21 MB 00:02 2024-04-22T00:59:14.031 INFO:teuthology.orchestra.run.smithi111.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-04-22T00:59:14.064 INFO:teuthology.orchestra.run.smithi203.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 1.3 MB/s | 222 kB 00:00 2024-04-22T00:59:14.123 INFO:teuthology.orchestra.run.smithi111.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 475 kB/s | 43 kB 00:00 2024-04-22T00:59:14.147 INFO:teuthology.orchestra.run.smithi203.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 325 kB/s | 27 kB 00:00 2024-04-22T00:59:14.165 INFO:teuthology.orchestra.run.smithi111.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-04-22T00:59:14.216 INFO:teuthology.orchestra.run.smithi111.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-04-22T00:59:14.231 INFO:teuthology.orchestra.run.smithi203.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 380 kB/s | 32 kB 00:00 2024-04-22T00:59:14.258 INFO:teuthology.orchestra.run.smithi111.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-22T00:59:14.316 INFO:teuthology.orchestra.run.smithi111.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-22T00:59:14.341 INFO:teuthology.orchestra.run.smithi111.stdout:(118/150): protobuf-compiler-3.14.0-13.el9.x86_ 492 kB/s | 863 kB 00:01 2024-04-22T00:59:14.375 INFO:teuthology.orchestra.run.smithi111.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-22T00:59:14.417 INFO:teuthology.orchestra.run.smithi111.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-22T00:59:14.476 INFO:teuthology.orchestra.run.smithi111.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-04-22T00:59:14.567 INFO:teuthology.orchestra.run.smithi111.stdout:(122/150): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-22T00:59:14.593 INFO:teuthology.orchestra.run.smithi111.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-04-22T00:59:14.627 INFO:teuthology.orchestra.run.smithi111.stdout:(124/150): python3-colorama-0.4.4-12.el9.noarch 124 kB/s | 35 kB 00:00 2024-04-22T00:59:14.668 INFO:teuthology.orchestra.run.smithi111.stdout:(125/150): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-04-22T00:59:14.702 INFO:teuthology.orchestra.run.smithi111.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-04-22T00:59:14.731 INFO:teuthology.orchestra.run.smithi203.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 1.5 MB/s | 745 kB 00:00 2024-04-22T00:59:14.761 INFO:teuthology.orchestra.run.smithi111.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-04-22T00:59:14.786 INFO:teuthology.orchestra.run.smithi111.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-04-22T00:59:14.811 INFO:teuthology.orchestra.run.smithi111.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-22T00:59:14.886 INFO:teuthology.orchestra.run.smithi111.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-22T00:59:14.898 INFO:teuthology.orchestra.run.smithi203.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 983 kB/s | 164 kB 00:00 2024-04-22T00:59:14.912 INFO:teuthology.orchestra.run.smithi111.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-04-22T00:59:14.987 INFO:teuthology.orchestra.run.smithi111.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-04-22T00:59:15.029 INFO:teuthology.orchestra.run.smithi111.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-22T00:59:15.065 INFO:teuthology.orchestra.run.smithi203.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 762 kB/s | 127 kB 00:00 2024-04-22T00:59:15.089 INFO:teuthology.orchestra.run.smithi111.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 161 kB/s | 16 kB 00:00 2024-04-22T00:59:15.132 INFO:teuthology.orchestra.run.smithi111.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 879 kB/s | 90 kB 00:00 2024-04-22T00:59:15.207 INFO:teuthology.orchestra.run.smithi111.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-04-22T00:59:15.232 INFO:teuthology.orchestra.run.smithi111.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-22T00:59:15.266 INFO:teuthology.orchestra.run.smithi111.stdout:(138/150): python3-kubernetes-26.1.0-2.el9.noar 2.0 MB/s | 1.0 MB 00:00 2024-04-22T00:59:15.307 INFO:teuthology.orchestra.run.smithi111.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-22T00:59:15.333 INFO:teuthology.orchestra.run.smithi111.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-04-22T00:59:15.366 INFO:teuthology.orchestra.run.smithi111.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-04-22T00:59:15.408 INFO:teuthology.orchestra.run.smithi111.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-04-22T00:59:15.442 INFO:teuthology.orchestra.run.smithi111.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-22T00:59:15.449 INFO:teuthology.orchestra.run.smithi203.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 1.3 MB/s | 501 kB 00:00 2024-04-22T00:59:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 894 kB/s | 90 kB 00:00 2024-04-22T00:59:15.509 INFO:teuthology.orchestra.run.smithi111.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-22T00:59:15.543 INFO:teuthology.orchestra.run.smithi111.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-22T00:59:15.668 INFO:teuthology.orchestra.run.smithi111.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 955 kB/s | 192 kB 00:00 2024-04-22T00:59:15.710 INFO:teuthology.orchestra.run.smithi111.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-04-22T00:59:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:(149/150): librados2-19.0.0-2405.g820a2f8c.el9. 1.9 MB/s | 3.4 MB 00:01 2024-04-22T00:59:17.457 INFO:teuthology.orchestra.run.smithi203.stdout:(18/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 11 MB/s | 48 MB 00:04 2024-04-22T00:59:17.566 INFO:teuthology.orchestra.run.smithi203.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 417 kB/s | 45 kB 00:00 2024-04-22T00:59:17.733 INFO:teuthology.orchestra.run.smithi203.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 799 kB/s | 133 kB 00:00 2024-04-22T00:59:17.825 INFO:teuthology.orchestra.run.smithi203.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 1.7 MB/s | 161 kB 00:00 2024-04-22T00:59:17.934 INFO:teuthology.orchestra.run.smithi203.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 2.9 MB/s | 321 kB 00:00 2024-04-22T00:59:18.035 INFO:teuthology.orchestra.run.smithi203.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 2.9 MB/s | 298 kB 00:00 2024-04-22T00:59:18.093 INFO:teuthology.orchestra.run.smithi203.stdout:(24/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.7 MB/s | 99 kB 00:00 2024-04-22T00:59:18.185 INFO:teuthology.orchestra.run.smithi203.stdout:(25/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 933 kB/s | 86 kB 00:00 2024-04-22T00:59:18.911 INFO:teuthology.orchestra.run.smithi203.stdout:(26/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 4.3 MB/s | 3.1 MB 00:00 2024-04-22T00:59:18.994 INFO:teuthology.orchestra.run.smithi203.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-22T00:59:19.069 INFO:teuthology.orchestra.run.smithi203.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 423 kB/s | 32 kB 00:00 2024-04-22T00:59:19.153 INFO:teuthology.orchestra.run.smithi203.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.7 MB/s | 141 kB 00:00 2024-04-22T00:59:19.212 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-22T00:59:19.829 INFO:teuthology.orchestra.run.smithi203.stdout:(30/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 1.2 MB/s | 5.3 MB 00:04 2024-04-22T00:59:20.162 INFO:teuthology.orchestra.run.smithi203.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 3.5 MB/s | 3.5 MB 00:01 2024-04-22T00:59:20.254 INFO:teuthology.orchestra.run.smithi203.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.7 MB/s | 249 kB 00:00 2024-04-22T00:59:20.338 INFO:teuthology.orchestra.run.smithi203.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 591 kB/s | 49 kB 00:00 2024-04-22T00:59:20.438 INFO:teuthology.orchestra.run.smithi203.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 153 kB/s | 15 kB 00:00 2024-04-22T00:59:20.522 INFO:teuthology.orchestra.run.smithi203.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.1 MB/s | 264 kB 00:00 2024-04-22T00:59:20.822 INFO:teuthology.orchestra.run.smithi203.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 2.5 MB/s | 762 kB 00:00 2024-04-22T00:59:20.897 INFO:teuthology.orchestra.run.smithi203.stdout:(37/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 243 kB/s | 18 kB 00:00 2024-04-22T00:59:20.927 INFO:teuthology.orchestra.run.smithi111.stdout:(150/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 611 kB/s | 3.1 MB 00:05 2024-04-22T00:59:20.933 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-22T00:59:20.933 INFO:teuthology.orchestra.run.smithi111.stdout:Total 9.9 MB/s | 216 MB 00:21 2024-04-22T00:59:21.072 INFO:teuthology.orchestra.run.smithi203.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 459 kB/s | 80 kB 00:00 2024-04-22T00:59:21.190 INFO:teuthology.orchestra.run.smithi203.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-04-22T00:59:21.257 INFO:teuthology.orchestra.run.smithi203.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 195 kB 00:00 2024-04-22T00:59:21.324 INFO:teuthology.orchestra.run.smithi203.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-22T00:59:21.416 INFO:teuthology.orchestra.run.smithi203.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-22T00:59:21.465 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T00:59:21.465 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T00:59:21.474 INFO:teuthology.orchestra.run.smithi203.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-22T00:59:21.477 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T00:59:21.477 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T00:59:21.478 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T00:59:21.479 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T00:59:21.481 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T00:59:21.483 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T00:59:21.484 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T00:59:21.533 INFO:teuthology.orchestra.run.smithi203.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-22T00:59:21.548 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-22T00:59:21.558 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-22T00:59:21.559 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T00:59:21.560 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T00:59:21.561 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-22T00:59:21.562 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T00:59:21.563 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T00:59:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T00:59:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T00:59:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T00:59:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T00:59:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T00:59:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T00:59:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout:Install 148 Packages 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-04-22T00:59:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:59:21.572 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 216 M 2024-04-22T00:59:21.572 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-22T00:59:21.659 INFO:teuthology.orchestra.run.smithi203.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 997 kB/s | 125 kB 00:00 2024-04-22T00:59:21.726 INFO:teuthology.orchestra.run.smithi203.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-22T00:59:21.998 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-22T00:59:22.076 INFO:teuthology.orchestra.run.smithi203.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 689 kB 00:00 2024-04-22T00:59:22.124 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-22T00:59:22.125 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-22T00:59:22.309 INFO:teuthology.orchestra.run.smithi203.stdout:(48/150): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-04-22T00:59:22.368 INFO:teuthology.orchestra.run.smithi203.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-22T00:59:22.427 INFO:teuthology.orchestra.run.smithi203.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 367 kB/s | 21 kB 00:00 2024-04-22T00:59:22.472 INFO:teuthology.orchestra.run.smithi038.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-22T00:59:22.519 INFO:teuthology.orchestra.run.smithi203.stdout:(51/150): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-22T00:59:22.569 INFO:teuthology.orchestra.run.smithi203.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-04-22T00:59:23.011 INFO:teuthology.orchestra.run.smithi203.stdout:(53/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.8 MB/s | 3.0 MB 00:00 2024-04-22T00:59:23.061 INFO:teuthology.orchestra.run.smithi203.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-22T00:59:23.112 INFO:teuthology.orchestra.run.smithi203.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-22T00:59:23.170 INFO:teuthology.orchestra.run.smithi203.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-22T00:59:23.221 INFO:teuthology.orchestra.run.smithi203.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-04-22T00:59:23.321 INFO:teuthology.orchestra.run.smithi203.stdout:(58/150): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-22T00:59:23.413 INFO:teuthology.orchestra.run.smithi203.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-22T00:59:23.472 INFO:teuthology.orchestra.run.smithi203.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-04-22T00:59:23.530 INFO:teuthology.orchestra.run.smithi203.stdout:(61/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-04-22T00:59:23.589 INFO:teuthology.orchestra.run.smithi203.stdout:(62/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-22T00:59:23.648 INFO:teuthology.orchestra.run.smithi203.stdout:(63/150): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-22T00:59:23.706 INFO:teuthology.orchestra.run.smithi203.stdout:(64/150): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-04-22T00:59:23.827 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-22T00:59:23.828 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-22T00:59:24.081 INFO:teuthology.orchestra.run.smithi038.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 553 kB/s | 889 kB 00:01 2024-04-22T00:59:24.107 INFO:teuthology.orchestra.run.smithi203.stdout:(65/150): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-04-22T00:59:24.322 INFO:teuthology.orchestra.run.smithi038.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 599 kB/s | 145 kB 00:00 2024-04-22T00:59:24.366 INFO:teuthology.orchestra.run.smithi203.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-04-22T00:59:24.499 INFO:teuthology.orchestra.run.smithi203.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-22T00:59:24.591 INFO:teuthology.orchestra.run.smithi203.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-04-22T00:59:24.672 INFO:teuthology.orchestra.run.smithi203.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-22T00:59:24.984 INFO:teuthology.orchestra.run.smithi203.stdout:(70/150): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-04-22T00:59:25.059 INFO:teuthology.orchestra.run.smithi203.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-22T00:59:25.109 INFO:teuthology.orchestra.run.smithi203.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-22T00:59:25.293 INFO:teuthology.orchestra.run.smithi203.stdout:(73/150): ceph-mgr-diskprediction-local-19.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-04-22T00:59:25.426 INFO:teuthology.orchestra.run.smithi203.stdout:(74/150): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-04-22T00:59:25.485 INFO:teuthology.orchestra.run.smithi203.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-22T00:59:25.544 INFO:teuthology.orchestra.run.smithi203.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 438 kB/s | 26 kB 00:00 2024-04-22T00:59:25.586 INFO:teuthology.orchestra.run.smithi203.stdout:(77/150): python3-devel-3.9.18-3.el9.x86_64.rpm 834 kB/s | 244 kB 00:00 2024-04-22T00:59:25.611 INFO:teuthology.orchestra.run.smithi203.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-04-22T00:59:25.661 INFO:teuthology.orchestra.run.smithi203.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-04-22T00:59:25.672 INFO:teuthology.orchestra.run.smithi038.stdout:(4/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-22T00:59:25.787 INFO:teuthology.orchestra.run.smithi203.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-04-22T00:59:25.854 INFO:teuthology.orchestra.run.smithi203.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-22T00:59:25.962 INFO:teuthology.orchestra.run.smithi203.stdout:(82/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-22T00:59:25.988 INFO:teuthology.orchestra.run.smithi203.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-22T00:59:26.021 INFO:teuthology.orchestra.run.smithi203.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-22T00:59:26.047 INFO:teuthology.orchestra.run.smithi203.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 719 kB/s | 42 kB 00:00 2024-04-22T00:59:26.080 INFO:teuthology.orchestra.run.smithi203.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-22T00:59:26.083 INFO:teuthology.orchestra.run.smithi038.stdout:(5/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 1.3 MB/s | 2.4 MB 00:01 2024-04-22T00:59:26.105 INFO:teuthology.orchestra.run.smithi203.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-22T00:59:26.125 INFO:teuthology.orchestra.run.smithi038.stdout:(6/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-04-22T00:59:26.139 INFO:teuthology.orchestra.run.smithi203.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-04-22T00:59:26.164 INFO:teuthology.orchestra.run.smithi203.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 868 kB/s | 51 kB 00:00 2024-04-22T00:59:26.207 INFO:teuthology.orchestra.run.smithi203.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 795 kB/s | 54 kB 00:00 2024-04-22T00:59:26.257 INFO:teuthology.orchestra.run.smithi203.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-22T00:59:26.299 INFO:teuthology.orchestra.run.smithi203.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-22T00:59:26.358 INFO:teuthology.orchestra.run.smithi203.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T00:59:26.408 INFO:teuthology.orchestra.run.smithi203.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-04-22T00:59:26.459 INFO:teuthology.orchestra.run.smithi203.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-22T00:59:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-22T00:59:26.576 INFO:teuthology.orchestra.run.smithi203.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-04-22T00:59:26.626 INFO:teuthology.orchestra.run.smithi203.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-22T00:59:26.685 INFO:teuthology.orchestra.run.smithi203.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T00:59:26.760 INFO:teuthology.orchestra.run.smithi203.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-04-22T00:59:27.136 INFO:teuthology.orchestra.run.smithi203.stdout:(101/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-22T00:59:27.161 INFO:teuthology.orchestra.run.smithi203.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 56 kB/s | 22 kB 00:00 2024-04-22T00:59:27.369 INFO:teuthology.orchestra.run.smithi203.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-04-22T00:59:27.428 INFO:teuthology.orchestra.run.smithi203.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 5.1 MB/s | 309 kB 00:00 2024-04-22T00:59:27.478 INFO:teuthology.orchestra.run.smithi203.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-04-22T00:59:27.645 INFO:teuthology.orchestra.run.smithi203.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2024-04-22T00:59:27.696 INFO:teuthology.orchestra.run.smithi203.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-04-22T00:59:27.746 INFO:teuthology.orchestra.run.smithi203.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 975 kB/s | 49 kB 00:00 2024-04-22T00:59:27.788 INFO:teuthology.orchestra.run.smithi203.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-22T00:59:27.838 INFO:teuthology.orchestra.run.smithi203.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-22T00:59:27.872 INFO:teuthology.orchestra.run.smithi203.stdout:(111/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-04-22T00:59:27.914 INFO:teuthology.orchestra.run.smithi203.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-04-22T00:59:27.964 INFO:teuthology.orchestra.run.smithi203.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 872 kB/s | 43 kB 00:00 2024-04-22T00:59:28.014 INFO:teuthology.orchestra.run.smithi203.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-04-22T00:59:28.065 INFO:teuthology.orchestra.run.smithi203.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-22T00:59:28.084 INFO:teuthology.orchestra.run.smithi038.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 2.3 MB/s | 4.7 MB 00:01 2024-04-22T00:59:28.099 INFO:teuthology.orchestra.run.smithi203.stdout:(116/150): python3-asyncssh-2.13.2-5.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-04-22T00:59:28.124 INFO:teuthology.orchestra.run.smithi203.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-22T00:59:28.158 INFO:teuthology.orchestra.run.smithi203.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-22T00:59:28.183 INFO:teuthology.orchestra.run.smithi203.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-22T00:59:28.208 INFO:teuthology.orchestra.run.smithi203.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-04-22T00:59:28.233 INFO:teuthology.orchestra.run.smithi203.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-04-22T00:59:28.284 INFO:teuthology.orchestra.run.smithi203.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-22T00:59:28.309 INFO:teuthology.orchestra.run.smithi038.stdout:(8/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x86 7.7 MB/s | 17 MB 00:02 2024-04-22T00:59:28.342 INFO:teuthology.orchestra.run.smithi203.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 15 MB/s | 2.0 MB 00:00 2024-04-22T00:59:28.359 INFO:teuthology.orchestra.run.smithi203.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-22T00:59:28.367 INFO:teuthology.orchestra.run.smithi038.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 434 kB/s | 25 kB 00:00 2024-04-22T00:59:28.393 INFO:teuthology.orchestra.run.smithi203.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-04-22T00:59:28.419 INFO:teuthology.orchestra.run.smithi203.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-04-22T00:59:28.444 INFO:teuthology.orchestra.run.smithi203.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-04-22T00:59:28.460 INFO:teuthology.orchestra.run.smithi203.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 472 kB/s | 20 kB 00:00 2024-04-22T00:59:28.502 INFO:teuthology.orchestra.run.smithi203.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-04-22T00:59:28.519 INFO:teuthology.orchestra.run.smithi203.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 790 kB/s | 46 kB 00:00 2024-04-22T00:59:28.544 INFO:teuthology.orchestra.run.smithi203.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-04-22T00:59:28.562 INFO:teuthology.orchestra.run.smithi203.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-04-22T00:59:28.595 INFO:teuthology.orchestra.run.smithi203.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-22T00:59:28.620 INFO:teuthology.orchestra.run.smithi203.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-22T00:59:28.646 INFO:teuthology.orchestra.run.smithi203.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-22T00:59:28.671 INFO:teuthology.orchestra.run.smithi203.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-22T00:59:28.696 INFO:teuthology.orchestra.run.smithi203.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-22T00:59:28.722 INFO:teuthology.orchestra.run.smithi203.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-22T00:59:28.747 INFO:teuthology.orchestra.run.smithi203.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-22T00:59:28.772 INFO:teuthology.orchestra.run.smithi203.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-04-22T00:59:28.798 INFO:teuthology.orchestra.run.smithi203.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-22T00:59:28.823 INFO:teuthology.orchestra.run.smithi203.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-22T00:59:28.857 INFO:teuthology.orchestra.run.smithi203.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-22T00:59:28.882 INFO:teuthology.orchestra.run.smithi203.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-22T00:59:28.907 INFO:teuthology.orchestra.run.smithi203.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-22T00:59:28.933 INFO:teuthology.orchestra.run.smithi203.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-22T00:59:28.983 INFO:teuthology.orchestra.run.smithi203.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-22T00:59:29.736 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-22T00:59:29.813 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T00:59:29.846 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T00:59:29.889 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T00:59:30.222 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T00:59:30.339 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-22T00:59:30.460 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-22T00:59:30.498 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T00:59:30.572 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T00:59:30.632 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-22T00:59:30.641 INFO:teuthology.orchestra.run.smithi203.stdout:(148/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-04-22T00:59:30.661 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T00:59:30.687 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T00:59:30.841 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T00:59:30.892 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T00:59:30.920 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T00:59:31.024 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T00:59:31.055 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-22T00:59:31.067 INFO:teuthology.orchestra.run.smithi203.stdout:(149/150): librados2-19.0.0-2405.g820a2f8c.el9. 1.6 MB/s | 3.4 MB 00:02 2024-04-22T00:59:31.117 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-22T00:59:31.139 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T00:59:31.160 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T00:59:31.356 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T00:59:31.389 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T00:59:31.496 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T00:59:31.585 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T00:59:31.629 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T00:59:31.659 INFO:teuthology.orchestra.run.smithi038.stdout:(10/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el 2.9 MB/s | 10 MB 00:03 2024-04-22T00:59:31.672 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T00:59:31.747 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T00:59:31.760 INFO:teuthology.orchestra.run.smithi038.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 2.2 MB/s | 222 kB 00:00 2024-04-22T00:59:31.788 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T00:59:31.832 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T00:59:31.860 INFO:teuthology.orchestra.run.smithi038.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 271 kB/s | 27 kB 00:00 2024-04-22T00:59:31.864 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T00:59:31.892 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T00:59:31.961 INFO:teuthology.orchestra.run.smithi038.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 317 kB/s | 32 kB 00:00 2024-04-22T00:59:31.963 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T00:59:32.023 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T00:59:32.053 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T00:59:32.097 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T00:59:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T00:59:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-22T00:59:32.236 INFO:teuthology.orchestra.run.smithi038.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 2.6 MB/s | 745 kB 00:00 2024-04-22T00:59:32.251 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-22T00:59:32.280 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-22T00:59:32.344 INFO:teuthology.orchestra.run.smithi038.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.5 MB/s | 164 kB 00:00 2024-04-22T00:59:32.352 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T00:59:32.395 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T00:59:32.480 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T00:59:32.500 INFO:teuthology.orchestra.run.smithi203.stdout:(150/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x 796 kB/s | 17 MB 00:21 2024-04-22T00:59:32.506 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-22T00:59:32.506 INFO:teuthology.orchestra.run.smithi203.stdout:Total 8.5 MB/s | 216 MB 00:25 2024-04-22T00:59:32.595 INFO:teuthology.orchestra.run.smithi038.stdout:(16/150): ceph-common-19.0.0-2405.g820a2f8c.el9 2.0 MB/s | 21 MB 00:10 2024-04-22T00:59:32.620 INFO:teuthology.orchestra.run.smithi038.stdout:(17/150): librados-devel-19.0.0-2405.g820a2f8c. 461 kB/s | 127 kB 00:00 2024-04-22T00:59:32.625 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T00:59:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T00:59:32.820 INFO:teuthology.orchestra.run.smithi038.stdout:(18/150): libradosstriper1-19.0.0-2405.g820a2f8 2.2 MB/s | 501 kB 00:00 2024-04-22T00:59:32.921 INFO:teuthology.orchestra.run.smithi038.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 452 kB/s | 45 kB 00:00 2024-04-22T00:59:33.021 INFO:teuthology.orchestra.run.smithi038.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.3 MB/s | 133 kB 00:00 2024-04-22T00:59:33.113 INFO:teuthology.orchestra.run.smithi038.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 1.7 MB/s | 161 kB 00:00 2024-04-22T00:59:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T00:59:33.288 INFO:teuthology.orchestra.run.smithi038.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 1.8 MB/s | 321 kB 00:00 2024-04-22T00:59:33.319 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T00:59:33.347 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T00:59:33.390 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T00:59:33.428 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-22T00:59:33.461 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T00:59:33.463 INFO:teuthology.orchestra.run.smithi038.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 1.7 MB/s | 298 kB 00:00 2024-04-22T00:59:33.516 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-22T00:59:33.516 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T00:59:33.564 INFO:teuthology.orchestra.run.smithi038.stdout:(24/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 995 kB/s | 99 kB 00:00 2024-04-22T00:59:33.589 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T00:59:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T00:59:33.640 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-22T00:59:33.641 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-22T00:59:33.665 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T00:59:33.713 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T00:59:33.748 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T00:59:33.791 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T00:59:33.833 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T00:59:33.973 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T00:59:34.014 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T00:59:34.014 INFO:teuthology.orchestra.run.smithi038.stdout:(25/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 8.6 MB/s | 48 MB 00:05 2024-04-22T00:59:34.039 INFO:teuthology.orchestra.run.smithi038.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 180 kB/s | 86 kB 00:00 2024-04-22T00:59:34.043 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T00:59:34.085 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T00:59:34.181 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T00:59:34.206 INFO:teuthology.orchestra.run.smithi038.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.0 MB/s | 171 kB 00:00 2024-04-22T00:59:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T00:59:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T00:59:34.290 INFO:teuthology.orchestra.run.smithi038.stdout:(28/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 3.2 MB/s | 5.3 MB 00:01 2024-04-22T00:59:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T00:59:34.315 INFO:teuthology.orchestra.run.smithi038.stdout:(29/150): ceph-grafana-dashboards-19.0.0-2405.g 293 kB/s | 32 kB 00:00 2024-04-22T00:59:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T00:59:34.392 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T00:59:34.399 INFO:teuthology.orchestra.run.smithi038.stdout:(30/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 8.0 MB/s | 3.1 MB 00:00 2024-04-22T00:59:34.424 INFO:teuthology.orchestra.run.smithi038.stdout:(31/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.0 MB/s | 141 kB 00:00 2024-04-22T00:59:34.522 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T00:59:34.525 INFO:teuthology.orchestra.run.smithi038.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.4 MB/s | 249 kB 00:00 2024-04-22T00:59:34.558 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T00:59:34.585 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T00:59:34.620 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T00:59:34.633 INFO:teuthology.orchestra.run.smithi038.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 456 kB/s | 49 kB 00:00 2024-04-22T00:59:34.664 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T00:59:34.710 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T00:59:34.742 INFO:teuthology.orchestra.run.smithi038.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 142 kB/s | 15 kB 00:00 2024-04-22T00:59:34.750 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T00:59:34.780 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T00:59:34.814 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T00:59:34.847 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T00:59:34.889 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T00:59:34.923 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T00:59:34.925 INFO:teuthology.orchestra.run.smithi038.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 1.4 MB/s | 264 kB 00:00 2024-04-22T00:59:35.012 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T00:59:35.129 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T00:59:35.160 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T00:59:35.207 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T00:59:35.242 INFO:teuthology.orchestra.run.smithi038.stdout:(36/150): ceph-mgr-diskprediction-local-19.0.0- 8.8 MB/s | 7.4 MB 00:00 2024-04-22T00:59:35.276 INFO:teuthology.orchestra.run.smithi038.stdout:(37/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 2.1 MB/s | 762 kB 00:00 2024-04-22T00:59:35.307 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T00:59:35.318 INFO:teuthology.orchestra.run.smithi038.stdout:(38/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 243 kB/s | 18 kB 00:00 2024-04-22T00:59:35.349 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-22T00:59:35.349 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-22T00:59:35.362 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T00:59:35.402 INFO:teuthology.orchestra.run.smithi038.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 641 kB/s | 80 kB 00:00 2024-04-22T00:59:35.468 INFO:teuthology.orchestra.run.smithi038.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-22T00:59:35.536 INFO:teuthology.orchestra.run.smithi038.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-22T00:59:35.561 INFO:teuthology.orchestra.run.smithi038.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-04-22T00:59:35.644 INFO:teuthology.orchestra.run.smithi038.stdout:(43/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 2.6 MB/s | 3.5 MB 00:01 2024-04-22T00:59:35.661 INFO:teuthology.orchestra.run.smithi038.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-22T00:59:35.695 INFO:teuthology.orchestra.run.smithi038.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 7.7 MB/s | 1.2 MB 00:00 2024-04-22T00:59:35.720 INFO:teuthology.orchestra.run.smithi038.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-22T00:59:35.746 INFO:teuthology.orchestra.run.smithi038.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-22T00:59:35.813 INFO:teuthology.orchestra.run.smithi038.stdout:(48/150): python3-pycparser-2.20-6.el9.noarch.r 805 kB/s | 135 kB 00:00 2024-04-22T00:59:36.102 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T00:59:36.150 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T00:59:36.184 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T00:59:36.238 INFO:teuthology.orchestra.run.smithi038.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 251 kB/s | 107 kB 00:00 2024-04-22T00:59:36.287 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T00:59:36.346 INFO:teuthology.orchestra.run.smithi038.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 199 kB/s | 21 kB 00:00 2024-04-22T00:59:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T00:59:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T00:59:36.469 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T00:59:36.488 INFO:teuthology.orchestra.run.smithi038.stdout:(51/150): automake-1.16.2-8.el9.noarch.rpm 898 kB/s | 689 kB 00:00 2024-04-22T00:59:36.597 INFO:teuthology.orchestra.run.smithi038.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-22T00:59:36.622 INFO:teuthology.orchestra.run.smithi038.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 33 kB 00:00 2024-04-22T00:59:36.647 INFO:teuthology.orchestra.run.smithi038.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 1.0 MB/s | 313 kB 00:00 2024-04-22T00:59:36.648 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T00:59:36.675 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T00:59:36.723 INFO:teuthology.orchestra.run.smithi038.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-04-22T00:59:36.748 INFO:teuthology.orchestra.run.smithi038.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-22T00:59:36.753 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T00:59:36.840 INFO:teuthology.orchestra.run.smithi038.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-22T00:59:36.865 INFO:teuthology.orchestra.run.smithi038.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-04-22T00:59:37.082 INFO:teuthology.orchestra.run.smithi038.stdout:(59/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.2 MB/s | 3.0 MB 00:00 2024-04-22T00:59:37.108 INFO:teuthology.orchestra.run.smithi038.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-04-22T00:59:37.191 INFO:teuthology.orchestra.run.smithi038.stdout:(61/150): libtool-2.4.6-45.el9.x86_64.rpm 1.7 MB/s | 582 kB 00:00 2024-04-22T00:59:37.217 INFO:teuthology.orchestra.run.smithi038.stdout:(62/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-04-22T00:59:37.242 INFO:teuthology.orchestra.run.smithi038.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 354 kB/s | 47 kB 00:00 2024-04-22T00:59:37.301 INFO:teuthology.orchestra.run.smithi038.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-22T00:59:37.326 INFO:teuthology.orchestra.run.smithi038.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-22T00:59:37.351 INFO:teuthology.orchestra.run.smithi038.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-04-22T00:59:37.414 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T00:59:37.485 INFO:teuthology.orchestra.run.smithi038.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-22T00:59:37.534 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T00:59:37.752 INFO:teuthology.orchestra.run.smithi038.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 4.1 MB 00:00 2024-04-22T00:59:38.019 INFO:teuthology.orchestra.run.smithi038.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 500 kB/s | 133 kB 00:00 2024-04-22T00:59:38.736 INFO:teuthology.orchestra.run.smithi038.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 728 kB/s | 911 kB 00:01 2024-04-22T00:59:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T00:59:38.863 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T00:59:38.891 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T00:59:38.924 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T00:59:39.237 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T00:59:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-22T00:59:39.369 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-22T00:59:39.400 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-22T00:59:39.436 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T00:59:39.910 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T00:59:39.940 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-22T00:59:40.012 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-22T00:59:40.041 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-22T00:59:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 747 kB/s | 1.0 MB 00:01 2024-04-22T00:59:40.328 INFO:teuthology.orchestra.run.smithi038.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 420 kB/s | 87 kB 00:00 2024-04-22T00:59:40.478 INFO:teuthology.orchestra.run.smithi038.stdout:(73/150): openblas-openmp-0.3.21-2.el9.x86_64.r 1.5 MB/s | 4.7 MB 00:03 2024-04-22T00:59:40.828 INFO:teuthology.orchestra.run.smithi038.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 697 kB/s | 244 kB 00:00 2024-04-22T00:59:41.179 INFO:teuthology.orchestra.run.smithi038.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 708 kB/s | 248 kB 00:00 2024-04-22T00:59:41.287 INFO:teuthology.orchestra.run.smithi038.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-04-22T00:59:41.396 INFO:teuthology.orchestra.run.smithi038.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 177 kB/s | 19 kB 00:00 2024-04-22T00:59:41.432 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-22T00:59:41.495 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T00:59:41.528 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T00:59:41.571 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T00:59:41.672 INFO:teuthology.orchestra.run.smithi038.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 626 kB/s | 172 kB 00:00 2024-04-22T00:59:41.780 INFO:teuthology.orchestra.run.smithi038.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-04-22T00:59:41.904 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T00:59:42.021 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-22T00:59:42.126 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-22T00:59:42.156 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T00:59:42.230 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T00:59:42.273 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-22T00:59:42.302 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T00:59:42.328 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T00:59:42.346 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T00:59:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T00:59:42.481 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T00:59:42.534 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T00:59:42.562 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T00:59:42.666 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T00:59:42.687 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-22T00:59:42.750 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-22T00:59:42.781 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T00:59:42.800 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T00:59:42.919 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T00:59:42.983 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-22T00:59:43.014 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T00:59:43.047 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T00:59:43.075 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-22T00:59:43.076 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T00:59:43.076 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T00:59:43.076 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:59:43.109 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-22T00:59:43.156 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T00:59:43.259 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T00:59:43.296 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T00:59:43.331 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T00:59:43.405 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T00:59:43.447 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T00:59:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T00:59:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T00:59:43.560 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T00:59:43.629 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T00:59:43.682 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T00:59:43.713 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T00:59:43.748 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T00:59:43.825 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T00:59:43.859 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-22T00:59:43.902 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-22T00:59:43.931 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-22T00:59:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T00:59:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T00:59:44.130 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T00:59:44.264 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T00:59:44.324 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T00:59:44.857 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T00:59:44.937 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T00:59:44.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T00:59:45.016 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T00:59:45.055 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-22T00:59:45.088 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T00:59:45.144 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T00:59:45.217 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T00:59:45.247 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T00:59:45.284 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T00:59:45.331 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T00:59:45.375 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T00:59:45.419 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T00:59:45.461 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T00:59:45.547 INFO:teuthology.orchestra.run.smithi038.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 786 kB/s | 5.8 MB 00:07 2024-04-22T00:59:45.599 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T00:59:45.642 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T00:59:45.672 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T00:59:45.689 INFO:teuthology.orchestra.run.smithi038.stdout:(81/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.3 MB/s | 5.1 MB 00:03 2024-04-22T00:59:45.713 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T00:59:45.797 INFO:teuthology.orchestra.run.smithi038.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-04-22T00:59:45.818 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T00:59:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T00:59:45.904 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T00:59:45.914 INFO:teuthology.orchestra.run.smithi038.stdout:(83/150): python3-babel-2.9.1-2.el9.noarch.rpm 1.1 MB/s | 6.0 MB 00:05 2024-04-22T00:59:45.929 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T00:59:45.954 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T00:59:46.014 INFO:teuthology.orchestra.run.smithi038.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 423 kB/s | 42 kB 00:00 2024-04-22T00:59:46.038 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T00:59:46.040 INFO:teuthology.orchestra.run.smithi038.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 318 kB/s | 77 kB 00:00 2024-04-22T00:59:46.057 INFO:teuthology.orchestra.run.smithi038.stdout:(86/150): python3-oauthlib-3.1.1-5.el9.noarch.r 604 kB/s | 222 kB 00:00 2024-04-22T00:59:46.168 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T00:59:46.205 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T00:59:46.232 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T00:59:46.266 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T00:59:46.307 INFO:teuthology.orchestra.run.smithi038.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 594 kB/s | 159 kB 00:00 2024-04-22T00:59:46.310 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T00:59:46.357 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T00:59:46.396 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T00:59:46.426 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T00:59:46.432 INFO:teuthology.orchestra.run.smithi038.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 408 kB/s | 51 kB 00:00 2024-04-22T00:59:46.458 INFO:teuthology.orchestra.run.smithi038.stdout:(89/150): python3-protobuf-3.14.0-13.el9.noarch 608 kB/s | 269 kB 00:00 2024-04-22T00:59:46.460 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T00:59:46.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T00:59:46.533 INFO:teuthology.orchestra.run.smithi038.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 537 kB/s | 54 kB 00:00 2024-04-22T00:59:46.535 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T00:59:46.558 INFO:teuthology.orchestra.run.smithi038.stdout:(91/150): python3-pyasn1-modules-0.4.8-6.el9.no 556 kB/s | 279 kB 00:00 2024-04-22T00:59:46.570 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T00:59:46.633 INFO:teuthology.orchestra.run.smithi038.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-04-22T00:59:46.659 INFO:teuthology.orchestra.run.smithi038.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-04-22T00:59:46.659 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T00:59:46.701 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T00:59:46.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T00:59:46.759 INFO:teuthology.orchestra.run.smithi038.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 358 kB/s | 36 kB 00:00 2024-04-22T00:59:46.788 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T00:59:46.860 INFO:teuthology.orchestra.run.smithi038.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-22T00:59:46.896 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T00:59:46.951 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T00:59:46.977 INFO:teuthology.orchestra.run.smithi038.stdout:(96/150): qatlib-24.02.0-1.el9.x86_64.rpm 644 kB/s | 221 kB 00:00 2024-04-22T00:59:47.077 INFO:teuthology.orchestra.run.smithi038.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 580 kB/s | 58 kB 00:00 2024-04-22T00:59:47.269 INFO:teuthology.orchestra.run.smithi038.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 489 kB/s | 94 kB 00:00 2024-04-22T00:59:47.302 INFO:teuthology.orchestra.run.smithi038.stdout:(99/150): socat-1.7.4.1-5.el9.x86_64.rpm 689 kB/s | 305 kB 00:00 2024-04-22T00:59:47.411 INFO:teuthology.orchestra.run.smithi038.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 414 kB/s | 45 kB 00:00 2024-04-22T00:59:47.453 INFO:teuthology.orchestra.run.smithi038.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-04-22T00:59:47.478 INFO:teuthology.orchestra.run.smithi038.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-04-22T00:59:47.629 INFO:teuthology.orchestra.run.smithi038.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-04-22T00:59:47.687 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T00:59:47.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T00:59:47.773 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T00:59:47.837 INFO:teuthology.orchestra.run.smithi038.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-04-22T00:59:47.874 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T00:59:47.912 INFO:teuthology.orchestra.run.smithi038.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-04-22T00:59:47.932 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T00:59:47.955 INFO:teuthology.orchestra.run.smithi038.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 949 kB/s | 309 kB 00:00 2024-04-22T00:59:48.021 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T00:59:48.021 INFO:teuthology.orchestra.run.smithi038.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-04-22T00:59:48.059 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T00:59:48.105 INFO:teuthology.orchestra.run.smithi038.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-04-22T00:59:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:(109/150): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-22T00:59:48.214 INFO:teuthology.orchestra.run.smithi038.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-04-22T00:59:48.239 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T00:59:48.265 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T00:59:48.266 INFO:teuthology.orchestra.run.smithi038.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-22T00:59:48.340 INFO:teuthology.orchestra.run.smithi038.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-04-22T00:59:48.343 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T00:59:48.390 INFO:teuthology.orchestra.run.smithi038.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-04-22T00:59:48.415 INFO:teuthology.orchestra.run.smithi038.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 577 kB/s | 43 kB 00:00 2024-04-22T00:59:48.457 INFO:teuthology.orchestra.run.smithi038.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-04-22T00:59:48.483 INFO:teuthology.orchestra.run.smithi038.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-22T00:59:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-22T00:59:48.558 INFO:teuthology.orchestra.run.smithi038.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-22T00:59:48.608 INFO:teuthology.orchestra.run.smithi038.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-22T00:59:48.634 INFO:teuthology.orchestra.run.smithi038.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-04-22T00:59:48.684 INFO:teuthology.orchestra.run.smithi038.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-22T00:59:48.743 INFO:teuthology.orchestra.run.smithi038.stdout:(122/150): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-22T00:59:48.768 INFO:teuthology.orchestra.run.smithi038.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-22T00:59:48.810 INFO:teuthology.orchestra.run.smithi038.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-04-22T00:59:48.835 INFO:teuthology.orchestra.run.smithi038.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-04-22T00:59:48.877 INFO:teuthology.orchestra.run.smithi038.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-04-22T00:59:48.903 INFO:teuthology.orchestra.run.smithi038.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-04-22T00:59:48.945 INFO:teuthology.orchestra.run.smithi038.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-04-22T00:59:49.002 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T00:59:49.011 INFO:teuthology.orchestra.run.smithi038.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 696 kB/s | 46 kB 00:00 2024-04-22T00:59:49.079 INFO:teuthology.orchestra.run.smithi038.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-22T00:59:49.112 INFO:teuthology.orchestra.run.smithi038.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-04-22T00:59:49.115 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T00:59:49.154 INFO:teuthology.orchestra.run.smithi038.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-04-22T00:59:49.188 INFO:teuthology.orchestra.run.smithi038.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-22T00:59:49.221 INFO:teuthology.orchestra.run.smithi038.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 244 kB/s | 16 kB 00:00 2024-04-22T00:59:49.263 INFO:teuthology.orchestra.run.smithi038.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-22T00:59:49.288 INFO:teuthology.orchestra.run.smithi038.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-22T00:59:49.331 INFO:teuthology.orchestra.run.smithi038.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 458 kB/s | 31 kB 00:00 2024-04-22T00:59:49.364 INFO:teuthology.orchestra.run.smithi038.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-22T00:59:49.398 INFO:teuthology.orchestra.run.smithi038.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-04-22T00:59:49.440 INFO:teuthology.orchestra.run.smithi038.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-22T00:59:49.465 INFO:teuthology.orchestra.run.smithi038.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 757 kB/s | 51 kB 00:00 2024-04-22T00:59:49.516 INFO:teuthology.orchestra.run.smithi038.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-22T00:59:49.541 INFO:teuthology.orchestra.run.smithi038.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-22T00:59:49.600 INFO:teuthology.orchestra.run.smithi038.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-22T00:59:49.616 INFO:teuthology.orchestra.run.smithi038.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-22T00:59:49.667 INFO:teuthology.orchestra.run.smithi038.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-22T00:59:49.934 INFO:teuthology.orchestra.run.smithi038.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-22T00:59:50.243 INFO:teuthology.orchestra.run.smithi038.stdout:(148/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 10 MB/s | 3.1 MB 00:00 2024-04-22T00:59:50.372 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T00:59:50.451 INFO:teuthology.orchestra.run.smithi038.stdout:(149/150): librados2-19.0.0-2405.g820a2f8c.el9. 4.3 MB/s | 3.4 MB 00:00 2024-04-22T00:59:50.455 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T00:59:50.483 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T00:59:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T00:59:50.831 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T00:59:50.870 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-22T00:59:50.953 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-22T00:59:50.984 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-22T00:59:51.020 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T00:59:51.491 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T00:59:51.532 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-22T00:59:51.604 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-22T00:59:51.634 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-22T00:59:53.964 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T00:59:53.999 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T00:59:54.539 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-04-22T00:59:54.567 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:59:54.577 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-22T00:59:54.667 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-22T00:59:54.667 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T00:59:54.667 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T00:59:54.667 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T00:59:54.696 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-22T00:59:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-22T00:59:54.770 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-22T00:59:54.771 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T00:59:54.771 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T00:59:54.771 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:59:56.241 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-22T00:59:56.267 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-22T00:59:56.268 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T00:59:56.268 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T00:59:56.268 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T00:59:56.268 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T00:59:56.268 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:59:56.492 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-22T00:59:56.514 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-22T00:59:56.514 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T00:59:56.514 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T00:59:56.515 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T00:59:56.515 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T00:59:56.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:59:56.951 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-22T00:59:56.973 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-22T00:59:56.973 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T00:59:56.973 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T00:59:56.973 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T00:59:56.973 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T00:59:56.974 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T00:59:57.081 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T00:59:57.138 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T00:59:57.194 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T00:59:57.376 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T00:59:57.426 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T00:59:57.496 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T00:59:57.583 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T00:59:57.722 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T00:59:57.760 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T00:59:57.792 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T00:59:57.820 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T00:59:58.238 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-22T00:59:58.268 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-22T00:59:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-22T00:59:58.362 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-22T00:59:58.383 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-22T00:59:59.408 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-22T00:59:59.443 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-22T01:00:00.402 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-22T01:00:00.436 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-22T01:00:00.552 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-22T01:00:00.715 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-22T01:00:00.746 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-22T01:00:00.769 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-22T01:00:00.769 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:00.769 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T01:00:00.769 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:00:00.770 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:00:00.770 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:00.813 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-22T01:00:00.825 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-22T01:00:00.922 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-22T01:00:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-22T01:00:01.863 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T01:00:01.899 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-22T01:00:01.921 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-22T01:00:01.921 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:01.921 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T01:00:01.921 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:00:01.922 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:00:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:01.967 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-22T01:00:01.988 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-22T01:00:01.988 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:01.988 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T01:00:01.989 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:02.034 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-22T01:00:02.056 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-22T01:00:02.056 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:02.056 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T01:00:02.056 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:00:02.056 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:00:02.057 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:02.359 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-22T01:00:02.382 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-22T01:00:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T01:00:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:00:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:00:02.382 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:04.017 INFO:teuthology.orchestra.run.smithi038.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 948 kB/s | 16 MB 00:17 2024-04-22T01:00:04.020 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:00:04.020 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.1 MB/s | 216 MB 00:42 2024-04-22T01:00:05.102 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-22T01:00:05.241 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-22T01:00:05.241 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-22T01:00:06.242 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-22T01:00:06.243 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:06.411 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-22T01:00:06.426 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-22T01:00:06.426 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:06.426 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T01:00:06.426 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:06.871 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-22T01:00:06.912 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-22T01:00:06.953 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-22T01:00:06.985 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T01:00:07.012 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T01:00:07.062 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-22T01:00:07.062 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-22T01:00:07.191 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-22T01:00:07.233 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T01:00:07.264 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-22T01:00:07.824 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:07.869 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:07.889 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:07.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-22T01:00:07.948 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-22T01:00:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T01:00:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:00:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:00:07.948 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:08.182 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-22T01:00:08.203 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-22T01:00:08.203 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:08.204 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T01:00:08.204 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:00:08.204 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:00:08.204 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:08.641 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-22T01:00:08.662 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-22T01:00:08.663 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-22T01:00:08.664 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:08.664 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T01:00:08.664 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:00:08.664 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:00:08.664 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T01:00:08.771 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T01:00:08.789 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T01:00:08.828 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T01:00:08.870 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T01:00:08.883 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T01:00:08.903 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T01:00:08.903 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T01:00:08.987 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T01:00:08.987 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:09.070 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T01:00:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T01:00:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T01:00:09.291 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T01:00:09.423 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T01:00:09.476 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T01:00:09.517 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T01:00:09.545 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T01:00:09.963 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-22T01:00:09.994 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-22T01:00:10.025 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-22T01:00:10.095 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-22T01:00:10.133 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-22T01:00:11.171 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-22T01:00:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-22T01:00:12.180 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-22T01:00:12.211 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-22T01:00:12.312 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-22T01:00:12.475 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-22T01:00:12.505 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-22T01:00:12.527 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-22T01:00:12.528 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:12.528 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T01:00:12.528 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:00:12.528 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:00:12.528 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-22T01:00:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-22T01:00:12.682 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-22T01:00:12.738 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-22T01:00:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-22T01:00:13.020 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T01:00:13.045 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T01:00:13.089 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T01:00:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T01:00:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-22T01:00:13.646 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T01:00:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-22T01:00:13.692 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-22T01:00:13.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T01:00:13.714 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-22T01:00:13.714 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:13.714 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T01:00:13.714 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:00:13.715 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:00:13.715 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:13.759 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-22T01:00:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-22T01:00:13.781 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:13.781 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T01:00:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:13.790 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T01:00:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-22T01:00:13.832 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-22T01:00:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-22T01:00:13.849 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:13.849 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T01:00:13.849 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:00:13.849 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:00:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:13.860 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T01:00:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T01:00:14.053 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T01:00:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T01:00:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T01:00:14.152 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-22T01:00:14.175 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-22T01:00:14.175 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:14.175 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T01:00:14.175 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:00:14.175 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:00:14.175 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:14.241 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T01:00:14.270 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-22T01:00:14.333 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-22T01:00:14.355 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T01:00:14.377 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T01:00:14.573 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T01:00:14.605 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T01:00:14.717 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T01:00:14.814 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T01:00:14.854 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T01:00:14.890 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T01:00:14.929 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-22T01:00:14.930 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-22T01:00:14.931 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-22T01:00:14.932 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T01:00:14.933 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T01:00:14.934 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T01:00:14.935 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T01:00:14.936 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T01:00:14.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T01:00:14.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T01:00:14.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T01:00:14.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T01:00:14.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T01:00:14.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T01:00:14.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-22T01:00:14.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T01:00:14.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-22T01:00:14.966 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T01:00:15.013 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T01:00:15.082 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T01:00:15.115 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T01:00:15.142 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T01:00:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T01:00:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T01:00:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T01:00:15.347 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T01:00:15.427 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T01:00:15.458 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-22T01:00:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-22T01:00:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-22T01:00:15.598 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T01:00:15.654 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T01:00:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T01:00:15.891 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T01:00:15.950 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T01:00:16.527 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T01:00:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T01:00:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T01:00:16.699 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T01:00:16.738 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-22T01:00:16.770 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T01:00:16.828 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T01:00:16.911 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-22T01:00:16.945 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.946 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.947 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T01:00:16.948 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T01:00:16.949 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T01:00:16.950 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T01:00:16.951 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T01:00:16.952 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T01:00:16.953 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T01:00:16.954 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T01:00:16.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T01:00:16.956 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T01:00:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T01:00:16.958 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T01:00:16.958 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-22T01:00:16.958 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-22T01:00:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T01:00:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T01:00:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T01:00:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T01:00:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T01:00:17.300 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T01:00:17.310 DEBUG:teuthology.parallel:result is None 2024-04-22T01:00:17.349 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T01:00:17.379 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T01:00:17.422 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T01:00:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T01:00:17.562 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T01:00:17.603 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T01:00:17.628 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T01:00:17.652 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T01:00:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T01:00:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T01:00:17.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T01:00:17.922 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T01:00:17.965 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T01:00:18.009 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T01:00:18.056 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T01:00:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T01:00:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T01:00:18.159 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T01:00:18.193 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T01:00:18.235 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T01:00:18.269 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T01:00:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T01:00:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T01:00:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T01:00:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T01:00:18.609 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T01:00:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T01:00:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-22T01:00:18.765 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-22T01:00:18.814 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-22T01:00:18.846 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T01:00:18.873 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T01:00:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-22T01:00:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T01:00:19.143 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-22T01:00:19.556 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T01:00:19.605 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T01:00:19.639 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T01:00:19.724 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:19.748 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T01:00:19.766 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:19.786 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:19.807 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T01:00:19.899 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T01:00:19.933 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T01:00:20.119 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T01:00:20.147 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T01:00:20.219 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T01:00:20.566 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-22T01:00:20.654 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T01:00:20.695 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T01:00:20.776 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T01:00:20.808 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T01:00:20.808 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T01:00:20.892 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T01:00:20.892 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:20.937 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T01:00:21.069 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T01:00:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T01:00:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T01:00:22.532 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T01:00:22.564 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T01:00:22.892 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T01:00:22.935 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-22T01:00:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-22T01:00:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-22T01:00:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T01:00:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T01:00:23.622 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-22T01:00:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-22T01:00:23.715 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-22T01:00:26.062 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T01:00:26.095 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T01:00:26.666 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-22T01:00:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-22T01:00:26.801 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-22T01:00:26.801 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T01:00:26.801 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T01:00:26.802 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:26.834 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-22T01:00:26.930 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-22T01:00:26.931 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-22T01:00:26.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-22T01:00:26.944 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T01:00:26.945 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T01:00:26.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T01:00:26.947 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T01:00:26.948 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T01:00:26.949 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T01:00:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T01:00:26.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T01:00:26.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T01:00:29.080 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.081 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T01:00:29.082 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T01:00:29.083 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T01:00:29.084 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T01:00:29.085 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T01:00:29.086 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T01:00:29.087 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T01:00:29.088 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T01:00:29.089 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.090 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:00:29.091 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-22T01:00:29.447 DEBUG:teuthology.parallel:result is None 2024-04-22T01:00:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-22T01:00:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-22T01:00:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-22T01:00:39.008 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:39.008 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T01:00:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:40.536 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-22T01:00:40.563 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-22T01:00:40.563 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:40.563 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T01:00:40.563 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:00:40.563 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:00:40.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:40.795 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-22T01:00:40.819 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-22T01:00:40.819 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:40.819 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T01:00:40.819 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:00:40.819 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:00:40.819 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:41.279 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-22T01:00:41.303 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-22T01:00:41.303 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:41.303 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T01:00:41.303 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:00:41.303 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:00:41.303 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:41.410 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T01:00:41.468 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T01:00:41.532 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T01:00:41.737 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T01:00:41.782 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T01:00:41.862 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T01:00:41.952 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T01:00:42.105 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T01:00:42.148 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T01:00:42.180 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T01:00:42.207 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T01:00:42.662 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-22T01:00:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-22T01:00:42.739 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-22T01:00:42.804 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-22T01:00:42.829 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-22T01:00:43.924 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-22T01:00:43.956 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-22T01:00:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-22T01:00:44.998 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-22T01:00:45.108 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-22T01:00:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-22T01:00:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-22T01:00:45.326 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-22T01:00:45.326 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:45.326 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T01:00:45.326 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:00:45.326 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:00:45.326 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-22T01:00:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-22T01:00:45.481 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-22T01:00:45.535 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-22T01:00:46.479 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T01:00:46.512 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-22T01:00:46.562 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-22T01:00:46.562 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:46.562 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T01:00:46.562 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:00:46.563 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:00:46.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:46.588 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-22T01:00:46.611 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-22T01:00:46.611 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:46.611 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T01:00:46.611 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:46.672 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-22T01:00:46.696 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-22T01:00:46.696 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:46.696 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T01:00:46.696 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:00:46.696 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T01:00:46.696 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:47.006 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-22T01:00:47.030 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-22T01:00:47.031 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:00:47.031 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T01:00:47.031 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:00:47.031 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:00:47.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:00:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-22T01:00:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-22T01:00:51.885 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-22T01:00:51.916 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T01:00:51.944 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T01:00:52.145 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-22T01:00:52.182 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T01:00:52.211 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-22T01:00:52.815 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:52.851 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:52.872 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T01:00:53.718 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-22T01:00:53.798 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T01:00:53.849 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T01:00:53.950 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T01:00:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T01:00:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T01:00:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T01:00:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:00:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-22T01:00:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-22T01:00:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-22T01:00:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T01:00:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T01:00:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T01:00:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T01:00:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T01:00:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T01:00:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T01:00:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T01:00:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-22T01:01:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T01:01:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:01:01.893 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-22T01:01:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T01:01:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T01:01:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T01:01:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T01:01:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T01:01:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T01:01:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T01:01:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T01:01:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T01:01:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T01:01:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T01:01:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T01:01:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:01:01.904 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-22T01:01:02.306 DEBUG:teuthology.parallel:result is None 2024-04-22T01:01:02.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:02.458 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T01:01:02.481 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-22T01:01:02.481 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-22T01:01:02.482 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-22T01:01:02.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:02.746 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T01:01:02.768 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-22T01:01:02.768 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-22T01:01:02.768 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-22T01:01:02.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:02.908 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T01:01:02.934 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-22T01:01:02.935 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-22T01:01:02.935 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-22T01:01:02.937 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T01:01:02.937 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:01:02.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:01:02.964 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:01:02.964 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:01:02.990 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:01:02.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:01:03.020 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T01:01:03.020 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:01:03.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:01:03.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:01:03.109 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:01:03.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:01:03.133 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:01:03.196 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:01:03.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:01:03.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:01:03.292 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T01:01:03.292 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:01:03.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:01:03.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:01:03.381 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:01:03.381 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:01:03.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:01:03.467 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:01:03.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:01:03.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:01:03.561 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T01:01:03.561 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:01:03.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:01:03.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:01:03.653 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:01:03.653 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:01:03.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:01:03.739 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:01:03.739 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:01:03.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:01:03.835 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T01:01:03.950 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-22T01:01:03.950 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:03.951 INFO:tasks.cephadm:Cluster fsid is caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:01:03.951 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T01:01:03.951 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.38', 'mon.b': '172.21.15.111', 'mon.c': '172.21.15.203'} 2024-04-22T01:01:03.951 INFO:tasks.cephadm:First mon is mon.a on smithi038 2024-04-22T01:01:03.951 INFO:tasks.cephadm:First mgr is x 2024-04-22T01:01:03.952 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T01:01:03.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-22T01:01:03.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-04-22T01:01:04.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-04-22T01:01:04.028 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-22T01:01:04.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:04.174 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54945/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2405-g820a2f8c', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2405.g820a2f8c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-20 17:52:04.986240', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-04-22T01:01:04.415 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 820a2f8ced66152ed3d765c31444b8076ea28c6b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:01:04.417 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm 2024-04-22T01:01:04.417 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:05.087 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 22 01:01 /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:05.087 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:05.798 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 22 01:01 /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:05.798 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:06.875 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 22 01:01 /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:06.875 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:06.895 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:06.914 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T01:01:06.948 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b on all hosts... 2024-04-22T01:01:06.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-22T01:01:06.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-22T01:01:06.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-22T01:01:08.547 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-22T01:01:08.599 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-22T01:01:08.618 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:01:46.632 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:01:49.252 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T01:01:49.252 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:01:49.252 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-22T01:01:49.252 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-04-22T01:01:49.252 INFO:teuthology.orchestra.run.smithi111.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-22T01:01:49.252 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-22T01:01:49.253 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T01:02:05.896 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T01:02:05.896 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:02:05.897 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-22T01:02:05.897 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-04-22T01:02:05.897 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-22T01:02:05.897 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-04-22T01:02:05.897 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T01:02:05.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-22T01:02:05.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-04-22T01:02:05.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-04-22T01:02:06.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-22T01:02:06.043 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-04-22T01:02:06.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-04-22T01:02:06.092 INFO:tasks.cephadm:Writing seed config... 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-22T01:02:06.094 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T01:02:06.095 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T01:02:06.096 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T01:02:06.097 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T01:02:06.097 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T01:02:06.097 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T01:02:06.097 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T01:02:06.097 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-22T01:02:06.098 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:02:06.098 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T01:02:06.115 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = caf31b2e-0043-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T01:02:06.116 DEBUG:teuthology.orchestra.run.smithi038:mon.a> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T01:02:06.158 DEBUG:teuthology.orchestra.run.smithi038:mgr.x> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T01:02:06.200 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T01:02:06.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b -v bootstrap --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:02:06.407 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:02:06.407 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b', '-v', 'bootstrap', '--fsid', 'caf31b2e-0043-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2024-04-22T01:02:06.407 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T01:02:06.408 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2024-04-22T01:02:06.434 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T01:02:06.434 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2024-04-22T01:02:06.434 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2024-04-22T01:02:06.442 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T01:02:06.442 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T01:02:06.449 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T01:02:06.449 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T01:02:06.456 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-04-22T01:02:06.462 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-04-22T01:02:06.463 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-04-22T01:02:06.463 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2024-04-22T01:02:06.720 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T01:02:06.720 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T01:02:06.721 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2024-04-22T01:02:06.721 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2024-04-22T01:02:06.727 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T01:02:06.727 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T01:02:06.734 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T01:02:06.734 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T01:02:06.741 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 139941740790688 on /run/cephadm/caf31b2e-0043-11ef-bc93-c7b262605968.lock 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 139941740790688 acquired on /run/cephadm/caf31b2e-0043-11ef-bc93-c7b262605968.lock 2024-04-22T01:02:06.748 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 3300 ... 2024-04-22T01:02:06.749 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2024-04-22T01:02:06.749 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-04-22T01:02:06.752 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-04-22T01:02:06.753 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-04-22T01:02:06.755 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T01:02:06.755 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-22T01:02:06.755 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-22T01:02:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T01:02:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T01:02:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T01:02:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-22T01:02:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-22T01:02:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T01:02:06.759 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-04-22T01:02:06.759 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-04-22T01:02:06.759 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T01:02:06.759 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T01:02:06.760 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-22T01:02:09.692 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-22T01:02:09.692 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-22T01:02:09.693 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T01:02:09.693 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:4706d97462bc6e0886c6ef3c0b1ac67e3e6a7e63a606e29958996d1fc15dafd8 2024-04-22T01:02:09.693 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T01:02:09.693 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-22T01:02:09.693 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T01:02:12.812 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-22T01:02:12.813 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-22T01:02:12.813 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T01:02:13.711 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2024-04-22T01:02:13.711 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2024-04-22T01:02:14.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQAWtyVmm4daAhAA0PxMtJoBV8+xRnRsKmPbgg== 2024-04-22T01:02:15.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQAWtyVm/NBjOhAACIXs7cNCulpfZwhQcsWdTQ== 2024-04-22T01:02:16.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQAXtyVmv7JiNBAAdZ/YjZ3mbdqKeYAee3PWUw== 2024-04-22T01:02:16.575 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for a [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T01:02:17.917 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:02:17.918 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2024-04-22T01:02:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.a on 2024-04-22T01:02:19.085 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T01:02:19.232 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T01:02:19.432 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-caf31b2e-0043-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-caf31b2e-0043-11ef-bc93-c7b262605968.target. 2024-04-22T01:02:19.432 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-caf31b2e-0043-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-caf31b2e-0043-11ef-bc93-c7b262605968.target. 2024-04-22T01:02:19.662 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a 2024-04-22T01:02:19.662 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T01:02:19.833 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-caf31b2e-0043-11ef-bc93-c7b262605968.target.wants/ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-caf31b2e-0043-11ef-bc93-c7b262605968@.service. 2024-04-22T01:02:20.274 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:20 smithi038 podman[27639]: 2024-04-22 01:02:20.274460372 +0000 UTC m=+0.308324813 container init cc2b3e42230d5065e71b88a03bb0826d1621f3b3ddb7d73fc980ec672a361977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T01:02:20.398 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T01:02:20.398 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T01:02:20.405 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T01:02:20.405 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T01:02:20.405 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T01:02:20.405 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T01:02:20.405 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2024-04-22T01:02:20.405 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2024-04-22T01:02:20.536 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:20 smithi038 podman[27639]: 2024-04-22 01:02:20.277786577 +0000 UTC m=+0.311651022 container start cc2b3e42230d5065e71b88a03bb0826d1621f3b3ddb7d73fc980ec672a361977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T01:02:20.536 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:20 smithi038 ceph-mon[27659]: mkfs caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:20.536 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:20 smithi038 ceph-mon[27659]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T01:02:20.536 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:20 smithi038 bash[27639]: cc2b3e42230d5065e71b88a03bb0826d1621f3b3ddb7d73fc980ec672a361977 2024-04-22T01:02:20.536 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:20 smithi038 systemd[1]: Started Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.64783s) 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2024-04-22T01:02:21.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.889+0000 7fae99e66700 1 Processor -- start 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.889+0000 7fae99e66700 1 -- start start 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.889+0000 7fae99e66700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae9410aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.889+0000 7fae99e66700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae9407be30 con 0x7fae94108700 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.890+0000 7fae937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae9410aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.890+0000 7fae937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae9410aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54756/0 (socket says 172.21.15.38:54756) 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.890+0000 7fae937fe700 1 -- 172.21.15.38:0/817140114 learned_addr learned my addr 172.21.15.38:0/817140114 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.890+0000 7fae937fe700 1 -- 172.21.15.38:0/817140114 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9410b030 con 0x7fae94108700 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae937fe700 1 --2- 172.21.15.38:0/817140114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae9410aaf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae840098a0 tx=0x7fae84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230fc5e434d3872b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae927fc700 1 -- 172.21.15.38:0/817140114 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae84004160 con 0x7fae94108700 2024-04-22T01:02:21.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae927fc700 1 -- 172.21.15.38:0/817140114 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae84004300 con 0x7fae94108700 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae927fc700 1 -- 172.21.15.38:0/817140114 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae840036a0 con 0x7fae94108700 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae99e66700 1 -- 172.21.15.38:0/817140114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 msgr2=0x7fae9410aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae99e66700 1 --2- 172.21.15.38:0/817140114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae9410aaf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae840098a0 tx=0x7fae84009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae99e66700 1 -- 172.21.15.38:0/817140114 shutdown_connections 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae99e66700 1 --2- 172.21.15.38:0/817140114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae9410aaf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.891+0000 7fae99e66700 1 -- 172.21.15.38:0/817140114 >> 172.21.15.38:0/817140114 conn(0x7fae94102110 msgr2=0x7fae94104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.892+0000 7fae99e66700 1 -- 172.21.15.38:0/817140114 shutdown_connections 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.892+0000 7fae99e66700 1 -- 172.21.15.38:0/817140114 wait complete. 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.892+0000 7fae99e66700 1 Processor -- start 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.892+0000 7fae99e66700 1 -- start start 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.892+0000 7fae99e66700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae94107f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.892+0000 7fae99e66700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae9410b500 con 0x7fae94108700 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae94107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:21.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae94107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54764/0 (socket says 172.21.15.38:54764) 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae937fe700 1 -- 172.21.15.38:0/728919462 learned_addr learned my addr 172.21.15.38:0/728919462 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae937fe700 1 -- 172.21.15.38:0/728919462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae941084c0 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae937fe700 1 --2- 172.21.15.38:0/728919462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae94107f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fae84000c00 tx=0x7fae84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae84003db0 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae84011630 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae941066b0 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae84011950 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.893+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae94106bf0 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.894+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fae8401a460 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.894+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fae8401aa90 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.894+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae94063890 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.897+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fae8401ac90 con 0x7fae94108700 2024-04-22T01:02:21.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.971+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fae941949f0 con 0x7fae94108700 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.971+0000 7fae90ff9700 1 -- 172.21.15.38:0/728919462 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7fae8401b650 con 0x7fae94108700 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 msgr2=0x7fae94107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 --2- 172.21.15.38:0/728919462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae94107f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fae84000c00 tx=0x7fae84004750 comp rx=0 tx=0).stop 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 shutdown_connections 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 --2- 172.21.15.38:0/728919462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae94108700 0x7fae94107f80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 >> 172.21.15.38:0/728919462 conn(0x7fae94102110 msgr2=0x7fae94102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 shutdown_connections 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:20.972+0000 7fae99e66700 1 -- 172.21.15.38:0/728919462 wait complete. 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2024-04-22T01:02:21.535 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T01:02:21.651 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T01:02:21.651 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: monmap epoch 1 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: last_changed 2024-04-22T01:02:16.910765+0000 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: min_mon_release 19 (squid) 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: election_strategy: 1 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: fsmap 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: mgrmap e1: no daemons active 2024-04-22T01:02:21.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:21 smithi038 ceph-mon[27659]: from='client.? 172.21.15.38:0/728919462' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T01:02:22.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:22 smithi038 ceph-mon[27659]: from='client.? 172.21.15.38:0/2060389541' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T01:02:22.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:22 smithi038 ceph-mon[27659]: from='client.? 172.21.15.38:0/2060389541' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T01:02:22.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:22.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T01:02:22.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:22.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-04-22T01:02:22.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T01:02:22.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T01:02:22.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc5328ad700 1 Processor -- start 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc5328ad700 1 -- start start 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc5328ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c10ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc5328ad700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c07bdf0 con 0x7fc52c108740 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc52bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c10ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc52bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c10ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54778/0 (socket says 172.21.15.38:54778) 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.960+0000 7fc52bfff700 1 -- 172.21.15.38:0/4147409805 learned_addr learned my addr 172.21.15.38:0/4147409805 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc52bfff700 1 -- 172.21.15.38:0/4147409805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52c10b070 con 0x7fc52c108740 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc52bfff700 1 --2- 172.21.15.38:0/4147409805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c10ab30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0098a0 tx=0x7fc51c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319bcae70fdd91f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc52b7fe700 1 -- 172.21.15.38:0/4147409805 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c004160 con 0x7fc52c108740 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc52b7fe700 1 -- 172.21.15.38:0/4147409805 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc51c004300 con 0x7fc52c108740 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc52b7fe700 1 -- 172.21.15.38:0/4147409805 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c0036a0 con 0x7fc52c108740 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc5328ad700 1 -- 172.21.15.38:0/4147409805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 msgr2=0x7fc52c10ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc5328ad700 1 --2- 172.21.15.38:0/4147409805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c10ab30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0098a0 tx=0x7fc51c009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc5328ad700 1 -- 172.21.15.38:0/4147409805 shutdown_connections 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc5328ad700 1 --2- 172.21.15.38:0/4147409805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c10ab30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.961+0000 7fc5328ad700 1 -- 172.21.15.38:0/4147409805 >> 172.21.15.38:0/4147409805 conn(0x7fc52c102170 msgr2=0x7fc52c104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc5328ad700 1 -- 172.21.15.38:0/4147409805 shutdown_connections 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc5328ad700 1 -- 172.21.15.38:0/4147409805 wait complete. 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc5328ad700 1 Processor -- start 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc5328ad700 1 -- start start 2024-04-22T01:02:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc5328ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc5328ad700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c10b4e0 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc52bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc52bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54794/0 (socket says 172.21.15.38:54794) 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.962+0000 7fc52bfff700 1 -- 172.21.15.38:0/2060389541 learned_addr learned my addr 172.21.15.38:0/2060389541 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc52bfff700 1 -- 172.21.15.38:0/2060389541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52c108500 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc52bfff700 1 --2- 172.21.15.38:0/2060389541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c107fc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009fb0 tx=0x7fc51c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c003de0 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc51c011630 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52c1066f0 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c011930 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc52c106c30 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc51c011ad0 con 0x7fc52c108740 2024-04-22T01:02:22.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.963+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc51c01b450 con 0x7fc52c108740 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.964+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc52c063890 con 0x7fc52c108740 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:21.966+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc51c004260 con 0x7fc52c108740 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.036+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc52c194d50 con 0x7fc52c108740 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.039+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fc51c025480 con 0x7fc52c108740 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.040+0000 7fc529ffb700 1 -- 172.21.15.38:0/2060389541 <== mon.0 v2:172.21.15.38:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc51c025660 con 0x7fc52c108740 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 msgr2=0x7fc52c107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 --2- 172.21.15.38:0/2060389541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c107fc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009fb0 tx=0x7fc51c004750 comp rx=0 tx=0).stop 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 shutdown_connections 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 --2- 172.21.15.38:0/2060389541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc52c108740 0x7fc52c107fc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 >> 172.21.15.38:0/2060389541 conn(0x7fc52c102170 msgr2=0x7fc52c104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 shutdown_connections 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:22.041+0000 7fc5328ad700 1 -- 172.21.15.38:0/2060389541 wait complete. 2024-04-22T01:02:22.610 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2024-04-22T01:02:23.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:23 smithi038 ceph-mon[27659]: from='client.? 172.21.15.38:0/355044264' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.028+0000 7f24636a2700 1 Processor -- start 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.028+0000 7f24636a2700 1 -- start start 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f24636a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c10aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f24636a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f245c07be30 con 0x7f245c108700 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f246143e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c10aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f246143e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c10aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54808/0 (socket says 172.21.15.38:54808) 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f246143e700 1 -- 172.21.15.38:0/1332430044 learned_addr learned my addr 172.21.15.38:0/1332430044 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f246143e700 1 -- 172.21.15.38:0/1332430044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c10b030 con 0x7f245c108700 2024-04-22T01:02:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f246143e700 1 --2- 172.21.15.38:0/1332430044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c10aaf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24580098a0 tx=0x7f2458009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe68fac742ea5fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.029+0000 7f2453fff700 1 -- 172.21.15.38:0/1332430044 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2458004160 con 0x7f245c108700 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f2453fff700 1 -- 172.21.15.38:0/1332430044 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2458004300 con 0x7f245c108700 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f2453fff700 1 -- 172.21.15.38:0/1332430044 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2458011b50 con 0x7f245c108700 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 -- 172.21.15.38:0/1332430044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 msgr2=0x7f245c10aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 --2- 172.21.15.38:0/1332430044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c10aaf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24580098a0 tx=0x7f2458009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 -- 172.21.15.38:0/1332430044 shutdown_connections 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 --2- 172.21.15.38:0/1332430044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c10aaf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 -- 172.21.15.38:0/1332430044 >> 172.21.15.38:0/1332430044 conn(0x7f245c102110 msgr2=0x7f245c104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 -- 172.21.15.38:0/1332430044 shutdown_connections 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.030+0000 7f24636a2700 1 -- 172.21.15.38:0/1332430044 wait complete. 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f24636a2700 1 Processor -- start 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f24636a2700 1 -- start start 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f24636a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f24636a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f245c10c060 con 0x7f245c108700 2024-04-22T01:02:23.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f246143e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f246143e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c19bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54814/0 (socket says 172.21.15.38:54814) 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f246143e700 1 -- 172.21.15.38:0/355044264 learned_addr learned my addr 172.21.15.38:0/355044264 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.031+0000 7f246143e700 1 -- 172.21.15.38:0/355044264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c19c180 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f246143e700 1 --2- 172.21.15.38:0/355044264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c19bc40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2458009fb0 tx=0x7f2458011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24580038b0 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2458011e50 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245c19c410 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2458019420 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f245c19f100 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2458019ad0 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.032+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2458024730 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.033+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f245c063890 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.035+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2458043dd0 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.105+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f245c19f2d0 con 0x7f245c108700 2024-04-22T01:02:23.638 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.106+0000 7f24527fc700 1 -- 172.21.15.38:0/355044264 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f2458032030 con 0x7f245c108700 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 msgr2=0x7f245c19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 --2- 172.21.15.38:0/355044264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c19bc40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2458009fb0 tx=0x7f2458011bf0 comp rx=0 tx=0).stop 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 shutdown_connections 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 --2- 172.21.15.38:0/355044264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f245c108700 0x7f245c19bc40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 >> 172.21.15.38:0/355044264 conn(0x7f245c102110 msgr2=0x7f245c102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 shutdown_connections 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:23.107+0000 7f24636a2700 1 -- 172.21.15.38:0/355044264 wait complete. 2024-04-22T01:02:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2024-04-22T01:02:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:23 smithi038 systemd[1]: Stopping Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:02:24.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:23 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a[27653]: 2024-04-22T01:02:23.942+0000 7f8da4f76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:02:24.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:23 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a[27653]: 2024-04-22T01:02:23.942+0000 7f8da4f76700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T01:02:24.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 podman[27913]: 2024-04-22 01:02:24.106843337 +0000 UTC m=+0.299779562 container died cc2b3e42230d5065e71b88a03bb0826d1621f3b3ddb7d73fc980ec672a361977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T01:02:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 podman[27913]: 2024-04-22 01:02:24.457627572 +0000 UTC m=+0.650563797 container cleanup cc2b3e42230d5065e71b88a03bb0826d1621f3b3ddb7d73fc980ec672a361977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T01:02:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 bash[27913]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a 2024-04-22T01:02:25.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 podman[27925]: 2024-04-22 01:02:24.743060614 +0000 UTC m=+0.637301744 container remove cc2b3e42230d5065e71b88a03bb0826d1621f3b3ddb7d73fc980ec672a361977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T01:02:25.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T01:02:25.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 systemd[1]: Stopped Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:02:25.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:24 smithi038 systemd[1]: Starting Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:02:25.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 podman[28024]: 2024-04-22 01:02:25.09546662 +0000 UTC m=+0.101628715 container create 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:02:25.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 podman[28024]: 2024-04-22 01:02:25.012355446 +0000 UTC m=+0.018517511 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:02:25.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 podman[28024]: 2024-04-22 01:02:25.289289916 +0000 UTC m=+0.295451973 container init 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:02:25.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 podman[28024]: 2024-04-22 01:02:25.292218619 +0000 UTC m=+0.298380675 container start 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev), process ceph-mon, pid 2 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: pidfile_write: ignore empty --pid-file 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: load: jerasure load: lrc 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: RocksDB version: 7.9.2 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Git sha 0 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Compile date 2024-04-20 15:04:30 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: DB SUMMARY 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: DB Session ID: 8LXKQX7VYC4B2LAY21WC 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: CURRENT file: CURRENT 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T01:02:25.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104886 ; 2024-04-22T01:02:25.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.error_if_exists: 0 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.create_if_missing: 0 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.paranoid_checks: 1 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.env: 0x562e08ccbc60 2024-04-22T01:02:25.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.info_log: 0x562e0b495aa0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.statistics: (nil) 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.use_fsync: 0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_log_file_size: 0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.allow_fallocate: 1 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T01:02:25.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.use_direct_reads: 0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.db_log_dir: 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.wal_dir: 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.write_buffer_manager: 0x562e0b4681e0 2024-04-22T01:02:25.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.unordered_write: 0 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.row_cache: None 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.wal_filter: None 2024-04-22T01:02:25.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.two_write_queues: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.wal_compression: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.atomic_flush: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.log_readahead_size: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T01:02:25.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_background_jobs: 2 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_background_compactions: -1 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_subcompactions: 1 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T01:02:25.371 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_open_files: -1 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_background_flushes: -1 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Compression algorithms supported: 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kZSTD supported: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kXpressCompression supported: 0 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kZlibCompression supported: 1 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kSnappyCompression supported: 1 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kLZ4Compression supported: 1 2024-04-22T01:02:25.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: kBZip2Compression supported: 0 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.merge_operator: 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_filter: None 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562e0b495ba0) 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T01:02:25.373 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_type: 0 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_index_type: 0 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_shortening: 1 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: checksum: 4 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: no_block_cache: 0 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache: 0x562e0b499090 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_options: 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: capacity : 536870912 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_shard_bits : 4 2024-04-22T01:02:25.374 INFO:journalctl@ceph.mon.a.smithi038.stdout: strict_capacity_limit : 0 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_compressed: (nil) 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: persistent_cache: (nil) 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size: 4096 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size_deviation: 10 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_restart_interval: 16 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_block_restart_interval: 1 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: metadata_block_size: 4096 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: partition_filters: 0 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: use_delta_encoding: 1 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: filter_policy: bloomfilter 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: whole_key_filtering: 1 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: verify_compression: 0 2024-04-22T01:02:25.375 INFO:journalctl@ceph.mon.a.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: format_version: 5 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_index_compression: 1 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_align: 0 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: max_auto_readahead_size: 262144 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: prepopulate_block_cache: 0 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: initial_auto_readahead_size: 8192 2024-04-22T01:02:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T01:02:25.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T01:02:25.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T01:02:25.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression: NoCompression 2024-04-22T01:02:25.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T01:02:25.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T01:02:25.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T01:02:25.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.num_levels: 7 2024-04-22T01:02:25.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T01:02:25.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T01:02:25.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T01:02:25.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T01:02:25.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T01:02:25.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T01:02:25.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T01:02:25.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T01:02:25.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T01:02:25.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T01:02:25.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T01:02:25.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T01:02:25.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T01:02:25.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T01:02:25.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T01:02:25.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T01:02:25.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T01:02:25.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T01:02:25.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T01:02:25.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T01:02:25.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T01:02:25.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T01:02:25.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T01:02:25.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T01:02:25.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T01:02:25.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T01:02:25.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T01:02:25.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T01:02:25.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T01:02:25.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.table_properties_collectors: 2024-04-22T01:02:25.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.inplace_update_support: 0 2024-04-22T01:02:25.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T01:02:25.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T01:02:25.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T01:02:25.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T01:02:25.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.bloom_locality: 0 2024-04-22T01:02:25.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.max_successive_merges: 0 2024-04-22T01:02:25.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T01:02:25.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.ttl: 2592000 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T01:02:25.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.enable_blob_files: false 2024-04-22T01:02:25.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.min_blob_size: 0 2024-04-22T01:02:25.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T01:02:25.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T01:02:25.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T01:02:25.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T01:02:25.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T01:02:25.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T01:02:25.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T01:02:25.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T01:02:25.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T01:02:25.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T01:02:25.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0e373e16-299a-4b53-aab4-fb767faac2ce 2024-04-22T01:02:25.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713747745322768, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T01:02:25.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T01:02:25.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713747745323928, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97909, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14195, "raw_average_key_size": 49, "raw_value_size": 90196, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713747745, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0e373e16-299a-4b53-aab4-fb767faac2ce", "db_session_id": "8LXKQX7VYC4B2LAY21WC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T01:02:25.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713747745324010, "job": 1, "event": "recovery_finished"} 2024-04-22T01:02:25.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T01:02:25.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T01:02:25.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562e0b58a000 2024-04-22T01:02:25.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: DB pointer 0x562e0b574000 2024-04-22T01:02:25.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T01:02:25.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T01:02:25.392 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** DB Stats ** 2024-04-22T01:02:25.393 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T01:02:25.393 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T01:02:25.393 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T01:02:25.393 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T01:02:25.393 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T01:02:25.393 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T01:02:25.394 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T01:02:25.394 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-22T01:02:25.394 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-22T01:02:25.394 INFO:journalctl@ceph.mon.a.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T01:02:25.394 INFO:journalctl@ceph.mon.a.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T01:02:25.394 INFO:journalctl@ceph.mon.a.smithi038.stdout: L0 2/0 99.49 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:02:25.395 INFO:journalctl@ceph.mon.a.smithi038.stdout: Sum 2/0 99.49 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:02:25.395 INFO:journalctl@ceph.mon.a.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:02:25.395 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-22T01:02:25.395 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-22T01:02:25.395 INFO:journalctl@ceph.mon.a.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 124.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T01:02:25.396 INFO:journalctl@ceph.mon.a.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T01:02:25.397 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T01:02:25.397 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T01:02:25.397 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T01:02:25.397 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T01:02:25.397 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative compaction: 0.00 GB write, 16.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T01:02:25.397 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval compaction: 0.00 GB write, 16.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T01:02:25.398 INFO:journalctl@ceph.mon.a.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T01:02:25.398 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache BinnedLRUCache@0x562e0b499090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-22T01:02:25.398 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-22T01:02:25.398 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-22T01:02:25.398 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T01:02:25.399 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: starting mon.a rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:25.399 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???) e1 preinit fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:25.399 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).mds e1 new map 2024-04-22T01:02:25.399 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).mds e1 print_map 2024-04-22T01:02:25.399 INFO:journalctl@ceph.mon.a.smithi038.stdout: e1 2024-04-22T01:02:25.399 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T01:02:25.400 INFO:journalctl@ceph.mon.a.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T01:02:25.400 INFO:journalctl@ceph.mon.a.smithi038.stdout: legacy client fscid: -1 2024-04-22T01:02:25.400 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-22T01:02:25.400 INFO:journalctl@ceph.mon.a.smithi038.stdout: No filesystems configured 2024-04-22T01:02:25.400 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T01:02:25.400 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T01:02:25.401 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T01:02:25.401 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T01:02:25.401 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T01:02:25.401 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 bash[28024]: 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a 2024-04-22T01:02:25.401 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 systemd[1]: Started Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:02:25.402 INFO:teuthology.orchestra.run.smithi038.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T01:02:25.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T01:02:25.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: monmap epoch 1 2024-04-22T01:02:25.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: last_changed 2024-04-22T01:02:16.910765+0000 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: min_mon_release 19 (squid) 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: election_strategy: 1 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: fsmap 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T01:02:25.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:25 smithi038 ceph-mon[28043]: mgrmap e1: no daemons active 2024-04-22T01:02:26.619 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.956+0000 7f19b1016700 1 Processor -- start 2024-04-22T01:02:26.619 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.956+0000 7f19b1016700 1 -- start start 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19b1016700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac1080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19b1016700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac108690 con 0x7f19ac107ca0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac1080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac1080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54826/0 (socket says 172.21.15.38:54826) 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19aad9d700 1 -- 172.21.15.38:0/2440342593 learned_addr learned my addr 172.21.15.38:0/2440342593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19aad9d700 1 -- 172.21.15.38:0/2440342593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac108e50 con 0x7f19ac107ca0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.957+0000 7f19aad9d700 1 --2- 172.21.15.38:0/2440342593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac1080c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f199c0098a0 tx=0x7f199c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f56019911b3026 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19a9d9b700 1 -- 172.21.15.38:0/2440342593 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f199c004160 con 0x7f19ac107ca0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19a9d9b700 1 -- 172.21.15.38:0/2440342593 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f199c004300 con 0x7f19ac107ca0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19a9d9b700 1 -- 172.21.15.38:0/2440342593 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f199c011b50 con 0x7f19ac107ca0 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 -- 172.21.15.38:0/2440342593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 msgr2=0x7f19ac1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 --2- 172.21.15.38:0/2440342593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac1080c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f199c0098a0 tx=0x7f199c009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 -- 172.21.15.38:0/2440342593 shutdown_connections 2024-04-22T01:02:26.620 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 --2- 172.21.15.38:0/2440342593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac1080c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 -- 172.21.15.38:0/2440342593 >> 172.21.15.38:0/2440342593 conn(0x7f19ac1032a0 msgr2=0x7f19ac1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 -- 172.21.15.38:0/2440342593 shutdown_connections 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.958+0000 7f19b1016700 1 -- 172.21.15.38:0/2440342593 wait complete. 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.959+0000 7f19b1016700 1 Processor -- start 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.959+0000 7f19b1016700 1 -- start start 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.959+0000 7f19b1016700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.959+0000 7f19b1016700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac10c200 con 0x7f19ac107ca0 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54832/0 (socket says 172.21.15.38:54832) 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19aad9d700 1 -- 172.21.15.38:0/4258915068 learned_addr learned my addr 172.21.15.38:0/4258915068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19aad9d700 1 -- 172.21.15.38:0/4258915068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac19c1d0 con 0x7f19ac107ca0 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19aad9d700 1 --2- 172.21.15.38:0/4258915068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac19bc90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f199c011850 tx=0x7f199c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f199c003af0 con 0x7f19ac107ca0 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f199c020070 con 0x7f19ac107ca0 2024-04-22T01:02:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac19c460 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f199c0195a0 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ac19f150 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.960+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f199c019c30 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.961+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f199c011e00 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.961+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ac063890 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:25.964+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f199c003c90 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.034+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f19ac19f320 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.038+0000 7f1993fff700 1 -- 172.21.15.38:0/4258915068 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f199c01a430 con 0x7f19ac107ca0 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 msgr2=0x7f19ac19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 --2- 172.21.15.38:0/4258915068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac19bc90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f199c011850 tx=0x7f199c011c10 comp rx=0 tx=0).stop 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 shutdown_connections 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 --2- 172.21.15.38:0/4258915068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19ac107ca0 0x7f19ac19bc90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 >> 172.21.15.38:0/4258915068 conn(0x7f19ac1032a0 msgr2=0x7f19ac103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:26.623 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 shutdown_connections 2024-04-22T01:02:26.623 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:26.039+0000 7f19b1016700 1 -- 172.21.15.38:0/4258915068 wait complete. 2024-04-22T01:02:26.623 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T01:02:26.644 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:02:26.644 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2024-04-22T01:02:26.645 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T01:02:26.645 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T01:02:26.917 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x 2024-04-22T01:02:26.918 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-22T01:02:27.077 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-caf31b2e-0043-11ef-bc93-c7b262605968.target.wants/ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-caf31b2e-0043-11ef-bc93-c7b262605968@.service. 2024-04-22T01:02:27.314 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:27 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4258915068' entity='client.admin' 2024-04-22T01:02:27.314 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 systemd[1]: Starting Ceph mgr.x for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:02:27.624 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 podman[28258]: 2024-04-22 01:02:27.314684173 +0000 UTC m=+0.112130882 container create 4798ca6bfa468bf17b5ac6f1c57afa91ebe89391a9ff00c4a02a941e9a6dd138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T01:02:27.624 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 podman[28258]: 2024-04-22 01:02:27.220892398 +0000 UTC m=+0.018339082 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-22T01:02:27.624 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 podman[28258]: 2024-04-22 01:02:27.525165971 +0000 UTC m=+0.322612636 container init 4798ca6bfa468bf17b5ac6f1c57afa91ebe89391a9ff00c4a02a941e9a6dd138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:02:27.625 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 podman[28258]: 2024-04-22 01:02:27.527644316 +0000 UTC m=+0.325090980 container start 4798ca6bfa468bf17b5ac6f1c57afa91ebe89391a9ff00c4a02a941e9a6dd138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T01:02:27.637 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T01:02:27.638 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T01:02:27.645 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T01:02:27.645 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T01:02:27.645 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T01:02:27.645 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T01:02:27.653 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T01:02:27.654 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T01:02:27.661 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T01:02:27.661 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T01:02:27.661 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T01:02:27.661 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-22T01:02:27.661 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2024-04-22T01:02:27.662 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2024-04-22T01:02:27.875 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 bash[28258]: 4798ca6bfa468bf17b5ac6f1c57afa91ebe89391a9ff00c4a02a941e9a6dd138 2024-04-22T01:02:27.875 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 systemd[1]: Started Ceph mgr.x for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:02:27.876 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:27.682+0000 7f326f98f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:02:27.876 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:27.875+0000 7f326f98f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:02:28.154 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:27 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:27.983+0000 7f326f98f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:02:28.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:28 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2101149843' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T01:02:28.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:28 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:28.314+0000 7f326f98f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:02:28.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:28 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:28.403+0000 7f326f98f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:02:29.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:29.061 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T01:02:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T01:02:29.063 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T01:02:29.064 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T01:02:29.071 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T01:02:29.071 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T01:02:29.071 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T01:02:29.071 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.071 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T01:02:29.072 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T01:02:20.326295+0000", 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T01:02:29.073 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.080+0000 7f55c2d9d700 1 Processor -- start 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.080+0000 7f55c2d9d700 1 -- start start 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.080+0000 7f55c2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.080+0000 7f55c2d9d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc106d00 con 0x7f55bc106310 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.081+0000 7f55c0b39700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.081+0000 7f55c0b39700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34370/0 (socket says 172.21.15.38:34370) 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.081+0000 7f55c0b39700 1 -- 172.21.15.38:0/3160928297 learned_addr learned my addr 172.21.15.38:0/3160928297 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.081+0000 7f55c0b39700 1 -- 172.21.15.38:0/3160928297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc107510 con 0x7f55bc106310 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.081+0000 7f55c0b39700 1 --2- 172.21.15.38:0/3160928297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc106730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f55b00098a0 tx=0x7f55b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56765abb8ecdced server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.081+0000 7f55bb7fe700 1 -- 172.21.15.38:0/3160928297 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b0004160 con 0x7f55bc106310 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55bb7fe700 1 -- 172.21.15.38:0/3160928297 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f55b0004300 con 0x7f55bc106310 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55bb7fe700 1 -- 172.21.15.38:0/3160928297 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b0011b50 con 0x7f55bc106310 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55c2d9d700 1 -- 172.21.15.38:0/3160928297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 msgr2=0x7f55bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55c2d9d700 1 --2- 172.21.15.38:0/3160928297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc106730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f55b00098a0 tx=0x7f55b0009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:29.074 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55c2d9d700 1 -- 172.21.15.38:0/3160928297 shutdown_connections 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55c2d9d700 1 --2- 172.21.15.38:0/3160928297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc106730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.082+0000 7f55c2d9d700 1 -- 172.21.15.38:0/3160928297 >> 172.21.15.38:0/3160928297 conn(0x7f55bc1018f0 msgr2=0x7f55bc103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.083+0000 7f55c2d9d700 1 -- 172.21.15.38:0/3160928297 shutdown_connections 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.083+0000 7f55c2d9d700 1 -- 172.21.15.38:0/3160928297 wait complete. 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.083+0000 7f55c2d9d700 1 Processor -- start 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.083+0000 7f55c2d9d700 1 -- start start 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.083+0000 7f55c2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.083+0000 7f55c2d9d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10a8c0 con 0x7f55bc106310 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c0b39700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c0b39700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34376/0 (socket says 172.21.15.38:34376) 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c0b39700 1 -- 172.21.15.38:0/2101149843 learned_addr learned my addr 172.21.15.38:0/2101149843 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c0b39700 1 -- 172.21.15.38:0/2101149843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc19ea00 con 0x7f55bc106310 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c0b39700 1 --2- 172.21.15.38:0/2101149843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc102e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009fb0 tx=0x7f55b0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b00038b0 con 0x7f55bc106310 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f55b0011e50 con 0x7f55bc106310 2024-04-22T01:02:29.075 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc19ebd0 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55b0019550 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.084+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc19eff0 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.085+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f55b0019b90 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.085+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f55b0024900 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.085+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55bc063890 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.088+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f55b0003a50 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.161+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f55bc02def0 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.162+0000 7f55b9ffb700 1 -- 172.21.15.38:0/2101149843 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f55b0032030 con 0x7f55bc106310 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.163+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 msgr2=0x7f55bc102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.163+0000 7f55c2d9d700 1 --2- 172.21.15.38:0/2101149843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc102e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009fb0 tx=0x7f55b0011bf0 comp rx=0 tx=0).stop 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.163+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 shutdown_connections 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.163+0000 7f55c2d9d700 1 --2- 172.21.15.38:0/2101149843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55bc106310 0x7f55bc102e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.163+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 >> 172.21.15.38:0/2101149843 conn(0x7f55bc1018f0 msgr2=0x7f55bc191a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.164+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 shutdown_connections 2024-04-22T01:02:29.076 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:28.164+0000 7f55c2d9d700 1 -- 172.21.15.38:0/2101149843 wait complete. 2024-04-22T01:02:29.077 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2024-04-22T01:02:30.011 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:29 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:29.662+0000 7f326f98f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:02:30.012 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:29 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:29.836+0000 7f326f98f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:02:30.012 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:30.011+0000 7f326f98f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:02:30.612 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:30.308+0000 7f326f98f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:02:30.612 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:30.402+0000 7f326f98f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:02:30.612 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:30.611+0000 7f326f98f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:02:30.895 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:30.808+0000 7f326f98f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:02:31.444 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:31 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:31.444+0000 7f326f98f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:02:31.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2687745893' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T01:02:32.154 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:31 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:31.897+0000 7f326f98f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:02:32.155 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:31 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:31.998+0000 7f326f98f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T01:02:32.278 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T01:02:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T01:02:32.281 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T01:02:20.326295+0000", 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:02:32.282 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66b270700 1 Processor -- start 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66b270700 1 -- start start 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66b270700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa664108090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66b270700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa664108660 con 0x7fa664107c70 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66900c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa664108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66900c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa664108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34378/0 (socket says 172.21.15.38:34378) 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.522+0000 7fa66900c700 1 -- 172.21.15.38:0/3636290974 learned_addr learned my addr 172.21.15.38:0/3636290974 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.523+0000 7fa66900c700 1 -- 172.21.15.38:0/3636290974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa664108e70 con 0x7fa664107c70 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.523+0000 7fa66900c700 1 --2- 172.21.15.38:0/3636290974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa664108090 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa6600098a0 tx=0x7fa660009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c07205c5fc17c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.523+0000 7fa65bfff700 1 -- 172.21.15.38:0/3636290974 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa660004160 con 0x7fa664107c70 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.523+0000 7fa65bfff700 1 -- 172.21.15.38:0/3636290974 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa660004300 con 0x7fa664107c70 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.523+0000 7fa65bfff700 1 -- 172.21.15.38:0/3636290974 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa660011b50 con 0x7fa664107c70 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 -- 172.21.15.38:0/3636290974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 msgr2=0x7fa664108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 --2- 172.21.15.38:0/3636290974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa664108090 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa6600098a0 tx=0x7fa660009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:32.283 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 -- 172.21.15.38:0/3636290974 shutdown_connections 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 --2- 172.21.15.38:0/3636290974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa664108090 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 -- 172.21.15.38:0/3636290974 >> 172.21.15.38:0/3636290974 conn(0x7fa664103210 msgr2=0x7fa664105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 -- 172.21.15.38:0/3636290974 shutdown_connections 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.524+0000 7fa66b270700 1 -- 172.21.15.38:0/3636290974 wait complete. 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66b270700 1 Processor -- start 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66b270700 1 -- start start 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66b270700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa66407daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66b270700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410c220 con 0x7fa664107c70 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66900c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa66407daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66900c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa66407daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34388/0 (socket says 172.21.15.38:34388) 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66900c700 1 -- 172.21.15.38:0/2687745893 learned_addr learned my addr 172.21.15.38:0/2687745893 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66900c700 1 -- 172.21.15.38:0/2687745893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66407e030 con 0x7fa664107c70 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.525+0000 7fa66900c700 1 --2- 172.21.15.38:0/2687745893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa66407daf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa660011850 tx=0x7fa660011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:32.284 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa660003af0 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa660020070 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6600195a0 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66407c220 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66407c760 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa660019ad0 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.526+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa660011e00 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.527+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa664063890 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.529+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa660003c90 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.602+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa66407ca50 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.603+0000 7fa65a7fc700 1 -- 172.21.15.38:0/2687745893 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa660025460 con 0x7fa664107c70 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.603+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 msgr2=0x7fa66407daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.603+0000 7fa66b270700 1 --2- 172.21.15.38:0/2687745893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa66407daf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa660011850 tx=0x7fa660011c10 comp rx=0 tx=0).stop 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.604+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 shutdown_connections 2024-04-22T01:02:32.285 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.604+0000 7fa66b270700 1 --2- 172.21.15.38:0/2687745893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa664107c70 0x7fa66407daf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:32.286 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.604+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 >> 172.21.15.38:0/2687745893 conn(0x7fa664103210 msgr2=0x7fa664103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:32.286 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.604+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 shutdown_connections 2024-04-22T01:02:32.286 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:31.604+0000 7fa66b270700 1 -- 172.21.15.38:0/2687745893 wait complete. 2024-04-22T01:02:32.286 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (2/15)... 2024-04-22T01:02:32.654 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:32.205+0000 7f326f98f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:02:33.097 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:32.748+0000 7f326f98f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:02:33.097 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:32.844+0000 7f326f98f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:02:33.404 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:33.097+0000 7f326f98f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:02:33.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:33.235+0000 7f326f98f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:02:33.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:33.335+0000 7f326f98f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:02:33.904 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:33.455+0000 7f326f98f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:02:33.905 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:33.549+0000 7f326f98f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:02:34.654 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:34.396+0000 7f326f98f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:02:34.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:34.621+0000 7f326f98f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: Activating manager daemon x 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: mgrmap e2: x(active, starting, since 0.00766411s) 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: Manager daemon x is now available 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' 2024-04-22T01:02:35.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' 2024-04-22T01:02:35.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:34 smithi038 ceph-mon[28043]: from='mgr.14100 172.21.15.38:0/733903771' entity='mgr.x' 2024-04-22T01:02:35.421 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T01:02:35.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T01:02:35.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T01:02:35.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T01:02:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:35.436 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T01:02:20.326295+0000", 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c031d0700 1 Processor -- start 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c031d0700 1 -- start start 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c031d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c031d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1086d0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c00f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c00f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34474/0 (socket says 172.21.15.38:34474) 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.743+0000 7f5c00f6c700 1 -- 172.21.15.38:0/337568529 learned_addr learned my addr 172.21.15.38:0/337568529 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5c00f6c700 1 -- 172.21.15.38:0/337568529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc108e40 con 0x7f5bfc107ce0 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5c00f6c700 1 --2- 172.21.15.38:0/337568529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc108100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0098a0 tx=0x7f5bec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fd4e6da85b594a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5bfb7fe700 1 -- 172.21.15.38:0/337568529 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bec004160 con 0x7f5bfc107ce0 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5bfb7fe700 1 -- 172.21.15.38:0/337568529 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5bec004300 con 0x7f5bfc107ce0 2024-04-22T01:02:35.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5bfb7fe700 1 -- 172.21.15.38:0/337568529 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bec011b50 con 0x7f5bfc107ce0 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5c031d0700 1 -- 172.21.15.38:0/337568529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 msgr2=0x7f5bfc108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.744+0000 7f5c031d0700 1 --2- 172.21.15.38:0/337568529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc108100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0098a0 tx=0x7f5bec009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 -- 172.21.15.38:0/337568529 shutdown_connections 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 --2- 172.21.15.38:0/337568529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc108100 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 -- 172.21.15.38:0/337568529 >> 172.21.15.38:0/337568529 conn(0x7f5bfc103300 msgr2=0x7f5bfc105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 -- 172.21.15.38:0/337568529 shutdown_connections 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 -- 172.21.15.38:0/337568529 wait complete. 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 Processor -- start 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 -- start start 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc19bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c031d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc10c1f0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c00f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc19bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c00f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc19bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34490/0 (socket says 172.21.15.38:34490) 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.745+0000 7f5c00f6c700 1 -- 172.21.15.38:0/2877460182 learned_addr learned my addr 172.21.15.38:0/2877460182 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5c00f6c700 1 -- 172.21.15.38:0/2877460182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc19c170 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5c00f6c700 1 --2- 172.21.15.38:0/2877460182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc19bc30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5bec000c00 tx=0x7f5bec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bec003b20 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5bec003cc0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bfc19c400 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bec024910 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.746+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bfc19f0f0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.747+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 2) ==== 46043+0+0 (secure 0 0 0) 0x7f5bec022020 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.747+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bfc063890 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.747+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5bec011e00 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.749+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5bec017ca0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.822+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5bfc19f2c0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.822+0000 7f5bf9ffb700 1 -- 172.21.15.38:0/2877460182 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5bfc19f2c0 con 0x7f5bfc107ce0 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 msgr2=0x7f5bfc19bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 --2- 172.21.15.38:0/2877460182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc19bc30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5bec000c00 tx=0x7f5bec011c10 comp rx=0 tx=0).stop 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 shutdown_connections 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 --2- 172.21.15.38:0/2877460182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc107ce0 0x7f5bfc19bc30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 >> 172.21.15.38:0/2877460182 conn(0x7f5bfc103300 msgr2=0x7f5bfc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 shutdown_connections 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:34.823+0000 7f5c031d0700 1 -- 172.21.15.38:0/2877460182 wait complete. 2024-04-22T01:02:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (3/15)... 2024-04-22T01:02:36.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:35 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2877460182' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T01:02:36.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:35 smithi038 ceph-mon[28043]: mgrmap e3: x(active, since 1.01321s) 2024-04-22T01:02:37.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:37 smithi038 ceph-mon[28043]: mgrmap e4: x(active, since 2s) 2024-04-22T01:02:38.773 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:38 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/147490763' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T01:02:38.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T01:02:38.815 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.816 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T01:02:38.816 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:38.816 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T01:02:38.816 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T01:02:38.816 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T01:02:38.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T01:02:38.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T01:02:38.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T01:02:38.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T01:02:38.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T01:02:38.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T01:02:38.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T01:02:38.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T01:02:38.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T01:02:38.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T01:02:38.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T01:02:38.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T01:02:38.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T01:02:38.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T01:02:38.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T01:02:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T01:02:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T01:02:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-22T01:02:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T01:02:38.834 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T01:02:38.834 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T01:02:20.326295+0000", 2024-04-22T01:02:38.834 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T01:02:38.834 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-22T01:02:38.834 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T01:02:38.834 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:02:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.883+0000 7f5912e9b700 1 Processor -- start 2024-04-22T01:02:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.883+0000 7f5912e9b700 1 -- start start 2024-04-22T01:02:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5912e9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c07c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5912e9b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f590c07c7a0 con 0x7f590c07ddb0 2024-04-22T01:02:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5910c37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c07c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5910c37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c07c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46970/0 (socket says 172.21.15.38:46970) 2024-04-22T01:02:38.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5910c37700 1 -- 172.21.15.38:0/3536630086 learned_addr learned my addr 172.21.15.38:0/3536630086 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:38.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5910c37700 1 -- 172.21.15.38:0/3536630086 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590c07c920 con 0x7f590c07ddb0 2024-04-22T01:02:38.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f5910c37700 1 --2- 172.21.15.38:0/3536630086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c07c260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f59000098a0 tx=0x7f5900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7e46b8dab17b5b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:38.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.884+0000 7f590b7fe700 1 -- 172.21.15.38:0/3536630086 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5900004160 con 0x7f590c07ddb0 2024-04-22T01:02:38.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f590b7fe700 1 -- 172.21.15.38:0/3536630086 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5900004300 con 0x7f590c07ddb0 2024-04-22T01:02:38.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f590b7fe700 1 -- 172.21.15.38:0/3536630086 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5900011b50 con 0x7f590c07ddb0 2024-04-22T01:02:38.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 -- 172.21.15.38:0/3536630086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 msgr2=0x7f590c07c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:38.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 --2- 172.21.15.38:0/3536630086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c07c260 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f59000098a0 tx=0x7f5900009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:38.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 -- 172.21.15.38:0/3536630086 shutdown_connections 2024-04-22T01:02:38.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 --2- 172.21.15.38:0/3536630086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c07c260 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:38.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 -- 172.21.15.38:0/3536630086 >> 172.21.15.38:0/3536630086 conn(0x7f590c103260 msgr2=0x7f590c105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:38.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 -- 172.21.15.38:0/3536630086 shutdown_connections 2024-04-22T01:02:38.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.885+0000 7f5912e9b700 1 -- 172.21.15.38:0/3536630086 wait complete. 2024-04-22T01:02:38.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5912e9b700 1 Processor -- start 2024-04-22T01:02:38.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5912e9b700 1 -- start start 2024-04-22T01:02:38.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5912e9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:38.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5912e9b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f590c10e250 con 0x7f590c07ddb0 2024-04-22T01:02:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5910c37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5910c37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46976/0 (socket says 172.21.15.38:46976) 2024-04-22T01:02:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5910c37700 1 -- 172.21.15.38:0/147490763 learned_addr learned my addr 172.21.15.38:0/147490763 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.886+0000 7f5910c37700 1 -- 172.21.15.38:0/147490763 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590c19e340 con 0x7f590c07ddb0 2024-04-22T01:02:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.887+0000 7f5910c37700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c19de00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5900011670 tx=0x7f5900011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.887+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59000039c0 con 0x7f590c07ddb0 2024-04-22T01:02:38.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.887+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5900011e50 con 0x7f590c07ddb0 2024-04-22T01:02:38.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.887+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590c19e5d0 con 0x7f590c07ddb0 2024-04-22T01:02:38.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.887+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f590001a5c0 con 0x7f590c07ddb0 2024-04-22T01:02:38.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.887+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f590c1a12c0 con 0x7f590c07ddb0 2024-04-22T01:02:38.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.888+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f590001a870 con 0x7f590c07ddb0 2024-04-22T01:02:38.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.888+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f590c063890 con 0x7f590c07ddb0 2024-04-22T01:02:38.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.888+0000 7f5909ffb700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f58fc039000 0x7f58fc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:38.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.888+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f590004d540 con 0x7f590c07ddb0 2024-04-22T01:02:38.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.888+0000 7f590bfff700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f58fc039000 0x7f58fc03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:38.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.889+0000 7f590bfff700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f58fc039000 0x7f58fc03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f58f8006fd0 tx=0x7f58f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:38.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:37.894+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5900019c70 con 0x7f590c07ddb0 2024-04-22T01:02:38.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.186+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f590c1a1490 con 0x7f590c07ddb0 2024-04-22T01:02:38.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.187+0000 7f5909ffb700 1 -- 172.21.15.38:0/147490763 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f590c1a1490 con 0x7f590c07ddb0 2024-04-22T01:02:38.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f58fc039000 msgr2=0x7f58fc03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:38.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f58fc039000 0x7f58fc03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f58f8006fd0 tx=0x7f58f8006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:38.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 msgr2=0x7f590c19de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:38.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c19de00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5900011670 tx=0x7f5900011b60 comp rx=0 tx=0).stop 2024-04-22T01:02:38.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 shutdown_connections 2024-04-22T01:02:38.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f58fc039000 0x7f58fc03b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:38.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 --2- 172.21.15.38:0/147490763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c07ddb0 0x7f590c19de00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:38.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 >> 172.21.15.38:0/147490763 conn(0x7f590c103260 msgr2=0x7f590c103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:38.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 shutdown_connections 2024-04-22T01:02:38.844 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:38.189+0000 7f5912e9b700 1 -- 172.21.15.38:0/147490763 wait complete. 2024-04-22T01:02:38.844 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2024-04-22T01:02:39.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:39 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1075721636' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T01:02:40.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T01:02:40.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.229+0000 7f76d51a5700 1 Processor -- start 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76d51a5700 1 -- start start 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76d51a5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d010ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76d51a5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f76d007be30 con 0x7f76d0108710 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76ced9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d010ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76ced9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d010ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46980/0 (socket says 172.21.15.38:46980) 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76ced9d700 1 -- 172.21.15.38:0/2301986778 learned_addr learned my addr 172.21.15.38:0/2301986778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76ced9d700 1 -- 172.21.15.38:0/2301986778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d010b040 con 0x7f76d0108710 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76ced9d700 1 --2- 172.21.15.38:0/2301986778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d010ab00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76b80098a0 tx=0x7f76b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1f17b5aa24ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.230+0000 7f76cdd9b700 1 -- 172.21.15.38:0/2301986778 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76b8004160 con 0x7f76d0108710 2024-04-22T01:02:40.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76cdd9b700 1 -- 172.21.15.38:0/2301986778 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f76b8004300 con 0x7f76d0108710 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76cdd9b700 1 -- 172.21.15.38:0/2301986778 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76b8011b50 con 0x7f76d0108710 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 -- 172.21.15.38:0/2301986778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 msgr2=0x7f76d010ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 --2- 172.21.15.38:0/2301986778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d010ab00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76b80098a0 tx=0x7f76b8009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 -- 172.21.15.38:0/2301986778 shutdown_connections 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 --2- 172.21.15.38:0/2301986778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d010ab00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 -- 172.21.15.38:0/2301986778 >> 172.21.15.38:0/2301986778 conn(0x7f76d0102160 msgr2=0x7f76d0104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 -- 172.21.15.38:0/2301986778 shutdown_connections 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.231+0000 7f76d51a5700 1 -- 172.21.15.38:0/2301986778 wait complete. 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.232+0000 7f76d51a5700 1 Processor -- start 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.232+0000 7f76d51a5700 1 -- start start 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.232+0000 7f76d51a5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d019bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.232+0000 7f76d51a5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f76d010c070 con 0x7f76d0108710 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.232+0000 7f76ced9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d019bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.232+0000 7f76ced9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d019bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46990/0 (socket says 172.21.15.38:46990) 2024-04-22T01:02:40.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76ced9d700 1 -- 172.21.15.38:0/1075721636 learned_addr learned my addr 172.21.15.38:0/1075721636 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76ced9d700 1 -- 172.21.15.38:0/1075721636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d019c190 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76ced9d700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d019bc50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009fb0 tx=0x7f76b8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76b80038b0 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d019c420 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f76b8011e50 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76b8019420 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.233+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d019f110 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.234+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f76b8019720 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.234+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d0063890 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.234+0000 7f76c7fff700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f76bc038fb0 0x7f76bc03b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.234+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f76b804d2a0 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.234+0000 7f76ce59c700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f76bc038fb0 0x7f76bc03b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.235+0000 7f76ce59c700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f76bc038fb0 0x7f76bc03b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.239+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f76b804cd00 con 0x7f76d0108710 2024-04-22T01:02:40.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.462+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f76d019f2e0 con 0x7f76d0108710 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.463+0000 7f76c7fff700 1 -- 172.21.15.38:0/1075721636 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f76d019f2e0 con 0x7f76d0108710 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f76bc038fb0 msgr2=0x7f76bc03b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f76bc038fb0 0x7f76bc03b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 msgr2=0x7f76d019bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d019bc50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009fb0 tx=0x7f76b8011bf0 comp rx=0 tx=0).stop 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 shutdown_connections 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f76bc038fb0 0x7f76bc03b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 --2- 172.21.15.38:0/1075721636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76d0108710 0x7f76d019bc50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 >> 172.21.15.38:0/1075721636 conn(0x7f76d0102160 msgr2=0x7f76d0102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 shutdown_connections 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:39.465+0000 7f76d51a5700 1 -- 172.21.15.38:0/1075721636 wait complete. 2024-04-22T01:02:40.121 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2024-04-22T01:02:41.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:40 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4052854322' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T01:02:41.129 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: ignoring --setuser ceph since I am not root 2024-04-22T01:02:41.129 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: ignoring --setgroup ceph since I am not root 2024-04-22T01:02:41.129 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:41.032+0000 7f87044bb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:02:41.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:41.224+0000 7f87044bb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:02:41.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:41.330+0000 7f87044bb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:02:41.457 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89dad72700 1 Processor -- start 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89dad72700 1 -- start start 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89dad72700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89dad72700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89d407bdd0 con 0x7f89d4108780 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89d8b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89d8b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47004/0 (socket says 172.21.15.38:47004) 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.527+0000 7f89d8b0e700 1 -- 172.21.15.38:0/4144641671 learned_addr learned my addr 172.21.15.38:0/4144641671 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:41.463 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.528+0000 7f89d8b0e700 1 -- 172.21.15.38:0/4144641671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d410b0b0 con 0x7f89d4108780 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.528+0000 7f89d8b0e700 1 --2- 172.21.15.38:0/4144641671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d410ab70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89c80098a0 tx=0x7f89c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85d5c8b315b3f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.528+0000 7f89d37fe700 1 -- 172.21.15.38:0/4144641671 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c8004160 con 0x7f89d4108780 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.528+0000 7f89d37fe700 1 -- 172.21.15.38:0/4144641671 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f89c8004300 con 0x7f89d4108780 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.528+0000 7f89d37fe700 1 -- 172.21.15.38:0/4144641671 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c8011b50 con 0x7f89d4108780 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 -- 172.21.15.38:0/4144641671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 msgr2=0x7f89d410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 --2- 172.21.15.38:0/4144641671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d410ab70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89c80098a0 tx=0x7f89c8009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 -- 172.21.15.38:0/4144641671 shutdown_connections 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 --2- 172.21.15.38:0/4144641671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d410ab70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 -- 172.21.15.38:0/4144641671 >> 172.21.15.38:0/4144641671 conn(0x7f89d4102170 msgr2=0x7f89d4104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 -- 172.21.15.38:0/4144641671 shutdown_connections 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 -- 172.21.15.38:0/4144641671 wait complete. 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 Processor -- start 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.529+0000 7f89dad72700 1 -- start start 2024-04-22T01:02:41.464 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89dad72700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89dad72700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89d410c080 con 0x7f89d4108780 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d8b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d8b0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d419bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47012/0 (socket says 172.21.15.38:47012) 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d8b0e700 1 -- 172.21.15.38:0/4052854322 learned_addr learned my addr 172.21.15.38:0/4052854322 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d8b0e700 1 -- 172.21.15.38:0/4052854322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d419c180 con 0x7f89d4108780 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d8b0e700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d419bc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009fb0 tx=0x7f89c8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c80038b0 con 0x7f89d4108780 2024-04-22T01:02:41.465 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f89c8011e50 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c8019420 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.530+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d419c410 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.531+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d419f100 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.531+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f89c8003a50 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.531+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d4063890 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.532+0000 7f89d1ffb700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f89c4039000 0x7f89c403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.532+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f89c804d120 con 0x7f89d4108780 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.532+0000 7f89d3fff700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f89c4039000 0x7f89c403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.532+0000 7f89d3fff700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f89c4039000 0x7f89c403b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f89c0006fd0 tx=0x7f89c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.537+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f89c801a4e0 con 0x7f89d4108780 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.801+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f89d419f2d0 con 0x7f89d4108780 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.853+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f89d419f2d0 con 0x7f89d4108780 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.854+0000 7f89d1ffb700 1 -- 172.21.15.38:0/4052854322 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f89c80196d0 con 0x7f89d4108780 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f89c4039000 msgr2=0x7f89c403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f89c4039000 0x7f89c403b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f89c0006fd0 tx=0x7f89c0006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 msgr2=0x7f89d419bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:41.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d419bc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009fb0 tx=0x7f89c8011bf0 comp rx=0 tx=0).stop 2024-04-22T01:02:41.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 shutdown_connections 2024-04-22T01:02:41.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f89c4039000 0x7f89c403b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:41.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 --2- 172.21.15.38:0/4052854322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89d4108780 0x7f89d419bc40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:41.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 >> 172.21.15.38:0/4052854322 conn(0x7f89d4102170 msgr2=0x7f89d4102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:41.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 shutdown_connections 2024-04-22T01:02:41.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:40.856+0000 7f89dad72700 1 -- 172.21.15.38:0/4052854322 wait complete. 2024-04-22T01:02:41.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:41.660+0000 7f87044bb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:02:41.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:41.750+0000 7f87044bb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:02:42.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4052854322' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T01:02:42.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:41 smithi038 ceph-mon[28043]: mgrmap e5: x(active, since 6s) 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:02:42.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.920+0000 7f26e1834700 1 Processor -- start 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.920+0000 7f26e1834700 1 -- start start 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.920+0000 7f26e1834700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.920+0000 7f26e1834700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc07bdf0 con 0x7f26dc106530 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47050/0 (socket says 172.21.15.38:47050) 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26daffd700 1 -- 172.21.15.38:0/3564085439 learned_addr learned my addr 172.21.15.38:0/3564085439 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26daffd700 1 -- 172.21.15.38:0/3564085439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc106e90 con 0x7f26dc106530 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26daffd700 1 --2- 172.21.15.38:0/3564085439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc106950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26c40098a0 tx=0x7f26c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b4e6cf9868a776 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26d9ffb700 1 -- 172.21.15.38:0/3564085439 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4004160 con 0x7f26dc106530 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26d9ffb700 1 -- 172.21.15.38:0/3564085439 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f26c4004300 con 0x7f26dc106530 2024-04-22T01:02:42.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.921+0000 7f26d9ffb700 1 -- 172.21.15.38:0/3564085439 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4011b50 con 0x7f26dc106530 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 -- 172.21.15.38:0/3564085439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 msgr2=0x7f26dc106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 --2- 172.21.15.38:0/3564085439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc106950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26c40098a0 tx=0x7f26c4009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 -- 172.21.15.38:0/3564085439 shutdown_connections 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 --2- 172.21.15.38:0/3564085439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc106950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 -- 172.21.15.38:0/3564085439 >> 172.21.15.38:0/3564085439 conn(0x7f26dc102110 msgr2=0x7f26dc104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 -- 172.21.15.38:0/3564085439 shutdown_connections 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.922+0000 7f26e1834700 1 -- 172.21.15.38:0/3564085439 wait complete. 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26e1834700 1 Processor -- start 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26e1834700 1 -- start start 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26e1834700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26e1834700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc107f20 con 0x7f26dc106530 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:42.840 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47058/0 (socket says 172.21.15.38:47058) 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.923+0000 7f26daffd700 1 -- 172.21.15.38:0/2431255600 learned_addr learned my addr 172.21.15.38:0/2431255600 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26daffd700 1 -- 172.21.15.38:0/2431255600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc19e380 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26daffd700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc19de40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26c4011b60 tx=0x7f26c4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4003970 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc19e610 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f26c4003b10 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4024610 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.924+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc1a1300 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.925+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f26c4024910 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.925+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26dc063890 con 0x7f26dc106530 2024-04-22T01:02:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.925+0000 7f26e0832700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 0x7f26c803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.925+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f26c4011e00 con 0x7f26dc106530 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.925+0000 7f26da7fc700 1 -- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 msgr2=0x7f26c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.926+0000 7f26da7fc700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 0x7f26c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:41.934+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f26c400b3e0 con 0x7f26dc106530 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.127+0000 7f26da7fc700 1 -- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 msgr2=0x7f26c803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.127+0000 7f26da7fc700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 0x7f26c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.246+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f26dc0008d0 con 0x7f26dc106530 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.247+0000 7f26e0832700 1 -- 172.21.15.38:0/2431255600 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f26c4016260 con 0x7f26dc106530 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 msgr2=0x7f26c803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:02:42.842 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 0x7f26c803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 msgr2=0x7f26dc19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc19de40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26c4011b60 tx=0x7f26c4011c40 comp rx=0 tx=0).stop 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 shutdown_connections 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7f26c8039050 0x7f26c803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 --2- 172.21.15.38:0/2431255600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26dc106530 0x7f26dc19de40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 >> 172.21.15.38:0/2431255600 conn(0x7f26dc102110 msgr2=0x7f26dc194a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 shutdown_connections 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:42.249+0000 7f26e1834700 1 -- 172.21.15.38:0/2431255600 wait complete. 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-04-22T01:02:42.843 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2024-04-22T01:02:43.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:42 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2431255600' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T01:02:43.085 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:43.026+0000 7f87044bb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:02:43.388 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:43.206+0000 7f87044bb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:02:43.388 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:43.387+0000 7f87044bb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:02:43.993 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:43.687+0000 7f87044bb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:02:43.993 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:43.783+0000 7f87044bb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:02:43.993 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:43.992+0000 7f87044bb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:02:44.404 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:44.192+0000 7f87044bb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:02:45.154 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:44.820+0000 7f87044bb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:02:45.567 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:45.259+0000 7f87044bb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:02:45.567 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:45.359+0000 7f87044bb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:02:45.567 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:45.567+0000 7f87044bb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:02:46.404 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.115+0000 7f87044bb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:02:46.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.211+0000 7f87044bb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:02:46.820 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.463+0000 7f87044bb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:02:46.820 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.601+0000 7f87044bb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:02:46.820 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.700+0000 7f87044bb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:02:46.820 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.819+0000 7f87044bb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:02:47.154 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:46.914+0000 7f87044bb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:02:48.037 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:47.760+0000 7f87044bb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:02:48.037 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:02:47.983+0000 7f87044bb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:02:48.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: Active manager daemon x restarted 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: Activating manager daemon x 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: mgrmap e6: x(active, starting, since 0.00860309s) 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: Manager daemon x is now available 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:48.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:48 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:49.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: Found migration_current of "None". Setting to last migration. 2024-04-22T01:02:49.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:02:49.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:02:49.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T01:02:49.300 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: [22/Apr/2024:01:02:48] ENGINE Bus STARTING 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: [22/Apr/2024:01:02:48] ENGINE Serving on http://172.21.15.38:8765 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: [22/Apr/2024:01:02:48] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: [22/Apr/2024:01:02:48] ENGINE Bus STARTED 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:02:49.301 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:49 smithi038 ceph-mon[28043]: mgrmap e7: x(active, since 1.00733s) 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd55e2c7700 1 Processor -- start 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd55e2c7700 1 -- start start 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd55e2c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55810ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd55e2c7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd55807bdf0 con 0x7fd558108740 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55810ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55810ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47068/0 (socket says 172.21.15.38:47068) 2024-04-22T01:02:49.579 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.319+0000 7fd557fff700 1 -- 172.21.15.38:0/2007692497 learned_addr learned my addr 172.21.15.38:0/2007692497 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.320+0000 7fd557fff700 1 -- 172.21.15.38:0/2007692497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55810b070 con 0x7fd558108740 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.320+0000 7fd557fff700 1 --2- 172.21.15.38:0/2007692497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55810ab30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5400098a0 tx=0x7fd540009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369a98f6e4f757b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.320+0000 7fd556ffd700 1 -- 172.21.15.38:0/2007692497 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd540004160 con 0x7fd558108740 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.320+0000 7fd556ffd700 1 -- 172.21.15.38:0/2007692497 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd540004300 con 0x7fd558108740 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.320+0000 7fd556ffd700 1 -- 172.21.15.38:0/2007692497 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd540011b50 con 0x7fd558108740 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 -- 172.21.15.38:0/2007692497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 msgr2=0x7fd55810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/2007692497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55810ab30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5400098a0 tx=0x7fd540009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 -- 172.21.15.38:0/2007692497 shutdown_connections 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/2007692497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55810ab30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 -- 172.21.15.38:0/2007692497 >> 172.21.15.38:0/2007692497 conn(0x7fd558102110 msgr2=0x7fd558104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:49.580 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 -- 172.21.15.38:0/2007692497 shutdown_connections 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 -- 172.21.15.38:0/2007692497 wait complete. 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.321+0000 7fd55e2c7700 1 Processor -- start 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd55e2c7700 1 -- start start 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd55e2c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd55e2c7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd55810c090 con 0x7fd558108740 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47084/0 (socket says 172.21.15.38:47084) 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd557fff700 1 -- 172.21.15.38:0/3811780310 learned_addr learned my addr 172.21.15.38:0/3811780310 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd557fff700 1 -- 172.21.15.38:0/3811780310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55819c180 con 0x7fd558108740 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.322+0000 7fd557fff700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55819bc40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd540011b60 tx=0x7fd540011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.323+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd540003970 con 0x7fd558108740 2024-04-22T01:02:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.323+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd540003b10 con 0x7fd558108740 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.323+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd540024610 con 0x7fd558108740 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.323+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55819c410 con 0x7fd558108740 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.323+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd55819f100 con 0x7fd558108740 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.324+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fd54001a9f0 con 0x7fd558108740 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.324+0000 7fd5557fa700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.324+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 --> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd54403bc70 con 0x7fd5440390a0 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.324+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd540011e00 con 0x7fd558108740 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.324+0000 7fd5577fe700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 msgr2=0x7fd54403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.324+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.525+0000 7fd5577fe700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 msgr2=0x7fd54403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:49.582 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.525+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.926+0000 7fd5577fe700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 msgr2=0x7fd54403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:43.926+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:44.727+0000 7fd5577fe700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 msgr2=0x7fd54403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:44.727+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:46.328+0000 7fd5577fe700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 msgr2=0x7fd54403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3674504753 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:46.328+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:47.992+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fd540024c40 con 0x7fd558108740 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:47.992+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 msgr2=0x7fd54403b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:47.992+0000 7fd5557fa700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.989+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fd54004ecb0 con 0x7fd558108740 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.989+0000 7fd5557fa700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd54403c9d0 0x7fd54403edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.989+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd54403bc70 con 0x7fd54403c9d0 2024-04-22T01:02:49.583 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.990+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd54403c9d0 0x7fd54403edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.990+0000 7fd5577fe700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd54403c9d0 0x7fd54403edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd548003a10 tx=0x7fd5480092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.991+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd54403bc70 con 0x7fd54403c9d0 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.998+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd558063890 con 0x7fd54403c9d0 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.998+0000 7fd5557fa700 1 -- 172.21.15.38:0/3811780310 <== mgr.14120 v2:172.21.15.38:6800/3063830265 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd558063890 con 0x7fd54403c9d0 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd54403c9d0 msgr2=0x7fd54403edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd54403c9d0 0x7fd54403edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd548003a10 tx=0x7fd5480092b0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 msgr2=0x7fd55819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55819bc40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd540011b60 tx=0x7fd540011c40 comp rx=0 tx=0).stop 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 shutdown_connections 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd54403c9d0 0x7fd54403edc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd558108740 0x7fd55819bc40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.584 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 --2- 172.21.15.38:0/3811780310 >> [v2:172.21.15.38:6800/3674504753,v1:172.21.15.38:6801/3674504753] conn(0x7fd5440390a0 0x7fd54403b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:49.585 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 >> 172.21.15.38:0/3811780310 conn(0x7fd558102110 msgr2=0x7fd558102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:49.585 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 shutdown_connections 2024-04-22T01:02:49.585 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:48.999+0000 7fd55e2c7700 1 -- 172.21.15.38:0/3811780310 wait complete. 2024-04-22T01:02:49.585 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2024-04-22T01:02:49.585 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2024-04-22T01:02:50.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:50 smithi038 ceph-mon[28043]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T01:02:50.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:50 smithi038 ceph-mon[28043]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T01:02:50.849 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f8494700 1 Processor -- start 2024-04-22T01:02:50.849 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f8494700 1 -- start start 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f8494700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f010abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f8494700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f00f007bd50 con 0x7f00f0108800 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f6230700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f010abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f6230700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f010abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57328/0 (socket says 172.21.15.38:57328) 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.023+0000 7f00f6230700 1 -- 172.21.15.38:0/4274428855 learned_addr learned my addr 172.21.15.38:0/4274428855 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f6230700 1 -- 172.21.15.38:0/4274428855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f010b130 con 0x7f00f0108800 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f6230700 1 --2- 172.21.15.38:0/4274428855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f010abf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0098a0 tx=0x7f00ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a504f57ebd116886 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f522e700 1 -- 172.21.15.38:0/4274428855 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00ec004160 con 0x7f00f0108800 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f522e700 1 -- 172.21.15.38:0/4274428855 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00ec004300 con 0x7f00f0108800 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f522e700 1 -- 172.21.15.38:0/4274428855 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00ec011b50 con 0x7f00f0108800 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f8494700 1 -- 172.21.15.38:0/4274428855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 msgr2=0x7f00f010abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.024+0000 7f00f8494700 1 --2- 172.21.15.38:0/4274428855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f010abf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0098a0 tx=0x7f00ec009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 -- 172.21.15.38:0/4274428855 shutdown_connections 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 --2- 172.21.15.38:0/4274428855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f010abf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 -- 172.21.15.38:0/4274428855 >> 172.21.15.38:0/4274428855 conn(0x7f00f0102170 msgr2=0x7f00f0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 -- 172.21.15.38:0/4274428855 shutdown_connections 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 -- 172.21.15.38:0/4274428855 wait complete. 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 Processor -- start 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 -- start start 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.025+0000 7f00f8494700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010c090 con 0x7f00f0108800 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f6230700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f6230700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57344/0 (socket says 172.21.15.38:57344) 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f6230700 1 -- 172.21.15.38:0/1917240564 learned_addr learned my addr 172.21.15.38:0/1917240564 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f6230700 1 -- 172.21.15.38:0/1917240564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f019c1d0 con 0x7f00f0108800 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f6230700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f019bc90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00ec011b60 tx=0x7f00ec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00ec003970 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00ec003b10 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f019c460 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00ec024610 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.026+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f019f150 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.027+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f00ec024910 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.027+0000 7f00e77fe700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f00dc038f30 0x7f00dc03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.027+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f00ec011e00 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.027+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f0063890 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.027+0000 7f00f5a2f700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f00dc038f30 0x7f00dc03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.028+0000 7f00f5a2f700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f00dc038f30 0x7f00dc03b3f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f00e0006fd0 tx=0x7f00e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.033+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f00ec02c350 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.055+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f00ec02da00 con 0x7f00f0108800 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.257+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f00f01033e0 con 0x7f00dc038f30 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.261+0000 7f00e77fe700 1 -- 172.21.15.38:0/1917240564 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f00f01033e0 con 0x7f00dc038f30 2024-04-22T01:02:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.263+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f00dc038f30 msgr2=0x7f00dc03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.263+0000 7f00f8494700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f00dc038f30 0x7f00dc03b3f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f00e0006fd0 tx=0x7f00e0006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.263+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 msgr2=0x7f00f019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.263+0000 7f00f8494700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f019bc90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00ec011b60 tx=0x7f00ec011c40 comp rx=0 tx=0).stop 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.264+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 shutdown_connections 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.264+0000 7f00f8494700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f00dc038f30 0x7f00dc03b3f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.264+0000 7f00f8494700 1 --2- 172.21.15.38:0/1917240564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00f0108800 0x7f00f019bc90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.264+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 >> 172.21.15.38:0/1917240564 conn(0x7f00f0102170 msgr2=0x7f00f0102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.264+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 shutdown_connections 2024-04-22T01:02:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:50.264+0000 7f00f8494700 1 -- 172.21.15.38:0/1917240564 wait complete. 2024-04-22T01:02:51.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:51 smithi038 ceph-mon[28043]: mgrmap e8: x(active, since 2s) 2024-04-22T01:02:51.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:51 smithi038 ceph-mon[28043]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:02:51.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:51 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:51.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:51 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febdb5d7700 1 Processor -- start 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febdb5d7700 1 -- start start 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febdb5d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd4106990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febdb5d7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7febd407bdb0 con 0x7febd4106570 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febd9373700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd4106990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febd9373700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd4106990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57348/0 (socket says 172.21.15.38:57348) 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.288+0000 7febd9373700 1 -- 172.21.15.38:0/828267773 learned_addr learned my addr 172.21.15.38:0/828267773 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.289+0000 7febd9373700 1 -- 172.21.15.38:0/828267773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd4106ed0 con 0x7febd4106570 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.289+0000 7febd9373700 1 --2- 172.21.15.38:0/828267773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd4106990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7febd00098a0 tx=0x7febd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4d1dce508e157b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.289+0000 7febcbfff700 1 -- 172.21.15.38:0/828267773 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febd0004160 con 0x7febd4106570 2024-04-22T01:02:52.106 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.289+0000 7febcbfff700 1 -- 172.21.15.38:0/828267773 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febd0004300 con 0x7febd4106570 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.289+0000 7febcbfff700 1 -- 172.21.15.38:0/828267773 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febd0011b50 con 0x7febd4106570 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 -- 172.21.15.38:0/828267773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 msgr2=0x7febd4106990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 --2- 172.21.15.38:0/828267773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd4106990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7febd00098a0 tx=0x7febd0009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 -- 172.21.15.38:0/828267773 shutdown_connections 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 --2- 172.21.15.38:0/828267773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd4106990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 -- 172.21.15.38:0/828267773 >> 172.21.15.38:0/828267773 conn(0x7febd4102170 msgr2=0x7febd4104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 -- 172.21.15.38:0/828267773 shutdown_connections 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 -- 172.21.15.38:0/828267773 wait complete. 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 Processor -- start 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.290+0000 7febdb5d7700 1 -- start start 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febdb5d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febdb5d7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7febd4107f60 con 0x7febd4106570 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febd9373700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febd9373700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57358/0 (socket says 172.21.15.38:57358) 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febd9373700 1 -- 172.21.15.38:0/4182859450 learned_addr learned my addr 172.21.15.38:0/4182859450 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febd9373700 1 -- 172.21.15.38:0/4182859450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd419e380 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febd9373700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd419de40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febd0011b60 tx=0x7febd0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febd0003970 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febd0003b10 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febd0024610 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd419e610 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.291+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd41a1300 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.292+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7febd00248c0 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.292+0000 7febca7fc700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7febc00390a0 0x7febc003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.292+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd4063890 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.293+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7febd0011e00 con 0x7febd4106570 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.293+0000 7febd8b72700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7febc00390a0 0x7febc003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.293+0000 7febd8b72700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7febc00390a0 0x7febc003b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febc4006fd0 tx=0x7febc4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.298+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7febd000bdb0 con 0x7febd4106570 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.506+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7febd402dea0 con 0x7febc00390a0 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.507+0000 7febca7fc700 1 -- 172.21.15.38:0/4182859450 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7febd402dea0 con 0x7febc00390a0 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7febc00390a0 msgr2=0x7febc003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7febc00390a0 0x7febc003b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febc4006fd0 tx=0x7febc4006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 msgr2=0x7febd419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd419de40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febd0011b60 tx=0x7febd0011c40 comp rx=0 tx=0).stop 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 shutdown_connections 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7febc00390a0 0x7febc003b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 --2- 172.21.15.38:0/4182859450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd4106570 0x7febd419de40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.509+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 >> 172.21.15.38:0/4182859450 conn(0x7febd4102170 msgr2=0x7febd4194a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.510+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 shutdown_connections 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:51.510+0000 7febdb5d7700 1 -- 172.21.15.38:0/4182859450 wait complete. 2024-04-22T01:02:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2024-04-22T01:02:52.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-mon[28043]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:02:53.065 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: Generating public/private rsa key pair. 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: Your identification has been saved in /tmp/tmppjruj099/key. 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: Your public key has been saved in /tmp/tmppjruj099/key.pub. 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: The key fingerprint is: 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: SHA256:Ws6hpeX9yHDYDEjZ+crZHiSkxw3AMB/UIKTFGY8qE20 ceph-caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: The key's randomart image is: 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: +---[RSA 3072]----+ 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | oB*=o | 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | . oo*.=.. | 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: |. E . = = | 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | o . . = + | 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: |o . o S + | 2024-04-22T01:02:53.066 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | o & & | 2024-04-22T01:02:53.067 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | + X B | 2024-04-22T01:02:53.067 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | = + | 2024-04-22T01:02:53.067 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: | + . | 2024-04-22T01:02:53.067 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:02:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: +----[SHA256]-----+ 2024-04-22T01:02:53.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:53 smithi038 ceph-mon[28043]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:02:53.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:53 smithi038 ceph-mon[28043]: Generating ssh key... 2024-04-22T01:02:53.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:53 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:53.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:53 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f38031d0700 1 Processor -- start 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f38031d0700 1 -- start start 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f38031d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f38031d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1086d0 con 0x7f37fc107ce0 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f3800f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f3800f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57368/0 (socket says 172.21.15.38:57368) 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.563+0000 7f3800f6c700 1 -- 172.21.15.38:0/343676971 learned_addr learned my addr 172.21.15.38:0/343676971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.564+0000 7f3800f6c700 1 -- 172.21.15.38:0/343676971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37fc108e40 con 0x7f37fc107ce0 2024-04-22T01:02:53.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.564+0000 7f3800f6c700 1 --2- 172.21.15.38:0/343676971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc108100 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37f00098a0 tx=0x7f37f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e170e231ee6b1b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.564+0000 7f37fb7fe700 1 -- 172.21.15.38:0/343676971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f0004160 con 0x7f37fc107ce0 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.564+0000 7f37fb7fe700 1 -- 172.21.15.38:0/343676971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f37f0004300 con 0x7f37fc107ce0 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.564+0000 7f37fb7fe700 1 -- 172.21.15.38:0/343676971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f0011b50 con 0x7f37fc107ce0 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 -- 172.21.15.38:0/343676971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 msgr2=0x7f37fc108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 --2- 172.21.15.38:0/343676971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc108100 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37f00098a0 tx=0x7f37f0009bb0 comp rx=0 tx=0).stop 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 -- 172.21.15.38:0/343676971 shutdown_connections 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 --2- 172.21.15.38:0/343676971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc108100 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 -- 172.21.15.38:0/343676971 >> 172.21.15.38:0/343676971 conn(0x7f37fc103300 msgr2=0x7f37fc105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 -- 172.21.15.38:0/343676971 shutdown_connections 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 -- 172.21.15.38:0/343676971 wait complete. 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.565+0000 7f38031d0700 1 Processor -- start 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f38031d0700 1 -- start start 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f38031d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc19bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f38031d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc10c1f0 con 0x7f37fc107ce0 2024-04-22T01:02:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f3800f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc19bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f3800f6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc19bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57378/0 (socket says 172.21.15.38:57378) 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f3800f6c700 1 -- 172.21.15.38:0/3490609012 learned_addr learned my addr 172.21.15.38:0/3490609012 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f3800f6c700 1 -- 172.21.15.38:0/3490609012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37fc19c170 con 0x7f37fc107ce0 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f3800f6c700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc19bc30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37f0011670 tx=0x7f37f0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.566+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f00039c0 con 0x7f37fc107ce0 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.567+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f37f0011e50 con 0x7f37fc107ce0 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.567+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37fc19c400 con 0x7f37fc107ce0 2024-04-22T01:02:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.567+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f001a5c0 con 0x7f37fc107ce0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.567+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37fc19f0f0 con 0x7f37fc107ce0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.568+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f37f001a870 con 0x7f37fc107ce0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.568+0000 7f37f9ffb700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f37ec039050 0x7f37ec03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.568+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f37f004d510 con 0x7f37fc107ce0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.568+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37fc063890 con 0x7f37fc107ce0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.568+0000 7f37fbfff700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f37ec039050 0x7f37ec03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.569+0000 7f37fbfff700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f37ec039050 0x7f37ec03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f37e8006fd0 tx=0x7f37e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.573+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f37f0019c70 con 0x7f37fc107ce0 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.784+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f37fc104540 con 0x7f37ec039050 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.953+0000 7f37f9ffb700 1 -- 172.21.15.38:0/3490609012 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f37fc104540 con 0x7f37ec039050 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f37ec039050 msgr2=0x7f37ec03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f37ec039050 0x7f37ec03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f37e8006fd0 tx=0x7f37e8006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 msgr2=0x7f37fc19bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc19bc30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37f0011670 tx=0x7f37f0011b60 comp rx=0 tx=0).stop 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 shutdown_connections 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f37ec039050 0x7f37ec03b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 --2- 172.21.15.38:0/3490609012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37fc107ce0 0x7f37fc19bc30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 >> 172.21.15.38:0/3490609012 conn(0x7f37fc103300 msgr2=0x7f37fc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 shutdown_connections 2024-04-22T01:02:53.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:52.956+0000 7f38031d0700 1 -- 172.21.15.38:0/3490609012 wait complete. 2024-04-22T01:02:54.835 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.060+0000 7fc613277700 1 Processor -- start 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.060+0000 7fc613277700 1 -- start start 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.060+0000 7fc613277700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c10a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.060+0000 7fc613277700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c10a820 con 0x7fc60c109ec0 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc611013700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c10a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc611013700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c10a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57384/0 (socket says 172.21.15.38:57384) 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc611013700 1 -- 172.21.15.38:0/2593397186 learned_addr learned my addr 172.21.15.38:0/2593397186 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc611013700 1 -- 172.21.15.38:0/2593397186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c10afd0 con 0x7fc60c109ec0 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc611013700 1 --2- 172.21.15.38:0/2593397186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c10a2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc604009e80 tx=0x7fc60400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdbe479137ca30a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc60bfff700 1 -- 172.21.15.38:0/2593397186 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604017070 con 0x7fc60c109ec0 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc60bfff700 1 -- 172.21.15.38:0/2593397186 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc60400b8c0 con 0x7fc60c109ec0 2024-04-22T01:02:54.836 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.061+0000 7fc60bfff700 1 -- 172.21.15.38:0/2593397186 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6040045c0 con 0x7fc60c109ec0 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 -- 172.21.15.38:0/2593397186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 msgr2=0x7fc60c10a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 --2- 172.21.15.38:0/2593397186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c10a2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc604009e80 tx=0x7fc60400b330 comp rx=0 tx=0).stop 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 -- 172.21.15.38:0/2593397186 shutdown_connections 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 --2- 172.21.15.38:0/2593397186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c10a2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 -- 172.21.15.38:0/2593397186 >> 172.21.15.38:0/2593397186 conn(0x7fc60c07d2b0 msgr2=0x7fc60c07d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 -- 172.21.15.38:0/2593397186 shutdown_connections 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 -- 172.21.15.38:0/2593397186 wait complete. 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.062+0000 7fc613277700 1 Processor -- start 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc613277700 1 -- start start 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc613277700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc613277700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c10e410 con 0x7fc60c109ec0 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc611013700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc611013700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57390/0 (socket says 172.21.15.38:57390) 2024-04-22T01:02:54.837 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc611013700 1 -- 172.21.15.38:0/2089808006 learned_addr learned my addr 172.21.15.38:0/2089808006 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc611013700 1 -- 172.21.15.38:0/2089808006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c19e330 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc611013700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c19ddf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc604009e50 tx=0x7fc60400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604020020 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.063+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc604003b50 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.064+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c19e5c0 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.064+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc604017040 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.064+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c1a12b0 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.064+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc604023020 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.065+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c063890 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.065+0000 7fc60a7fc700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc5f4039050 0x7fc5f403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.065+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc60400f050 con 0x7fc60c109ec0 2024-04-22T01:02:54.838 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.065+0000 7fc610812700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc5f4039050 0x7fc5f403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.065+0000 7fc610812700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc5f4039050 0x7fc5f403b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.070+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc60402a020 con 0x7fc60c109ec0 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.275+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fc60c107ae0 con 0x7fc5f4039050 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.276+0000 7fc60a7fc700 1 -- 172.21.15.38:0/2089808006 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fc60c107ae0 con 0x7fc5f4039050 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc5f4039050 msgr2=0x7fc5f403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc5f4039050 0x7fc5f403b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600006e40 comp rx=0 tx=0).stop 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 msgr2=0x7fc60c19ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c19ddf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc604009e50 tx=0x7fc60400bd30 comp rx=0 tx=0).stop 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 shutdown_connections 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc5f4039050 0x7fc5f403b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 --2- 172.21.15.38:0/2089808006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc60c109ec0 0x7fc60c19ddf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.278+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 >> 172.21.15.38:0/2089808006 conn(0x7fc60c07d2b0 msgr2=0x7fc60c1073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.279+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 shutdown_connections 2024-04-22T01:02:54.839 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:54.279+0000 7fc613277700 1 -- 172.21.15.38:0/2089808006 wait complete. 2024-04-22T01:02:54.840 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T01:02:54.840 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T01:02:54.840 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2024-04-22T01:02:55.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:54 smithi038 ceph-mon[28043]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:02:56.138 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:55 smithi038 ceph-mon[28043]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:02:57.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:02:56 smithi038 ceph-mon[28043]: Deploying cephadm binary to smithi038 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f90021ef700 1 Processor -- start 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f90021ef700 1 -- start start 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f90021ef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc10ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f90021ef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc07be30 con 0x7f8ffc108720 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f8ffb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc10ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f8ffb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc10ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57406/0 (socket says 172.21.15.38:57406) 2024-04-22T01:03:00.493 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.312+0000 7f8ffb7fe700 1 -- 172.21.15.38:0/2102120664 learned_addr learned my addr 172.21.15.38:0/2102120664 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f8ffb7fe700 1 -- 172.21.15.38:0/2102120664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc10b050 con 0x7f8ffc108720 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f8ffb7fe700 1 --2- 172.21.15.38:0/2102120664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc10ab10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0098a0 tx=0x7f8fec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448b468e155e5bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f8ffa7fc700 1 -- 172.21.15.38:0/2102120664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fec004160 con 0x7f8ffc108720 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f8ffa7fc700 1 -- 172.21.15.38:0/2102120664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8fec004300 con 0x7f8ffc108720 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f8ffa7fc700 1 -- 172.21.15.38:0/2102120664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fec011b50 con 0x7f8ffc108720 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f90021ef700 1 -- 172.21.15.38:0/2102120664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 msgr2=0x7f8ffc10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.313+0000 7f90021ef700 1 --2- 172.21.15.38:0/2102120664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc10ab10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0098a0 tx=0x7f8fec009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.314+0000 7f90021ef700 1 -- 172.21.15.38:0/2102120664 shutdown_connections 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.314+0000 7f90021ef700 1 --2- 172.21.15.38:0/2102120664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc10ab10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:00.494 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.314+0000 7f90021ef700 1 -- 172.21.15.38:0/2102120664 >> 172.21.15.38:0/2102120664 conn(0x7f8ffc102170 msgr2=0x7f8ffc104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.314+0000 7f90021ef700 1 -- 172.21.15.38:0/2102120664 shutdown_connections 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.314+0000 7f90021ef700 1 -- 172.21.15.38:0/2102120664 wait complete. 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.314+0000 7f90021ef700 1 Processor -- start 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f90021ef700 1 -- start start 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f90021ef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f90021ef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc10c080 con 0x7f8ffc108720 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f8ffb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f8ffb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57410/0 (socket says 172.21.15.38:57410) 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f8ffb7fe700 1 -- 172.21.15.38:0/2273119954 learned_addr learned my addr 172.21.15.38:0/2273119954 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f8ffb7fe700 1 -- 172.21.15.38:0/2273119954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ffc19c1c0 con 0x7f8ffc108720 2024-04-22T01:03:00.495 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.315+0000 7f8ffb7fe700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc19bc80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009fb0 tx=0x7f8fec011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.316+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fec0038b0 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.316+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8fec011e50 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.316+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ffc19c450 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.316+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fec019420 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.316+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ffc19f140 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.317+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8fec0196f0 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.317+0000 7f8ff8ff9700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f8fe4039050 0x7f8fe403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.317+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ffc063890 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.317+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8fec04d320 con 0x7f8ffc108720 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.317+0000 7f8ffaffd700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f8fe4039050 0x7f8fe403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:00.496 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.317+0000 7f8ffaffd700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f8fe4039050 0x7f8fe403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0006fd0 tx=0x7f8ff0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.322+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8fec00ba30 con 0x7f8ffc108720 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:55.533+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) -- 0x7f8ffc1033e0 con 0x7f8fe4039050 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.746+0000 7f8ff8ff9700 1 -- 172.21.15.38:0/2273119954 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8ffc1033e0 con 0x7f8fe4039050 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f8fe4039050 msgr2=0x7f8fe403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f8fe4039050 0x7f8fe403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0006fd0 tx=0x7f8ff0006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 msgr2=0x7f8ffc19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc19bc80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009fb0 tx=0x7f8fec011bf0 comp rx=0 tx=0).stop 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 shutdown_connections 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f8fe4039050 0x7f8fe403b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 --2- 172.21.15.38:0/2273119954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ffc108720 0x7f8ffc19bc80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:00.497 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 >> 172.21.15.38:0/2273119954 conn(0x7f8ffc102170 msgr2=0x7f8ffc102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:00.498 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 shutdown_connections 2024-04-22T01:03:00.498 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:02:59.748+0000 7f90021ef700 1 -- 172.21.15.38:0/2273119954 wait complete. 2024-04-22T01:03:00.498 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mon service... 2024-04-22T01:03:01.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:00 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:01.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:00 smithi038 ceph-mon[28043]: Added host smithi038 2024-04-22T01:03:01.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:00 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:02.027 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.009+0000 7fa699f74700 1 Processor -- start 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.009+0000 7fa699f74700 1 -- start start 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.009+0000 7fa699f74700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.009+0000 7fa699f74700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa69407bdf0 con 0x7fa694108740 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52526/0 (socket says 172.21.15.38:52526) 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6937fe700 1 -- 172.21.15.38:0/3122243886 learned_addr learned my addr 172.21.15.38:0/3122243886 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6937fe700 1 -- 172.21.15.38:0/3122243886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69410b070 con 0x7fa694108740 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6937fe700 1 --2- 172.21.15.38:0/3122243886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69410ab30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa684009e80 tx=0x7fa68400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87aee6a9b899c037 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6927fc700 1 -- 172.21.15.38:0/3122243886 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa684017070 con 0x7fa694108740 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6927fc700 1 -- 172.21.15.38:0/3122243886 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa68400b8c0 con 0x7fa694108740 2024-04-22T01:03:02.028 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.010+0000 7fa6927fc700 1 -- 172.21.15.38:0/3122243886 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6840045c0 con 0x7fa694108740 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 -- 172.21.15.38:0/3122243886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 msgr2=0x7fa69410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 --2- 172.21.15.38:0/3122243886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69410ab30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa684009e80 tx=0x7fa68400b330 comp rx=0 tx=0).stop 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 -- 172.21.15.38:0/3122243886 shutdown_connections 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 --2- 172.21.15.38:0/3122243886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69410ab30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 -- 172.21.15.38:0/3122243886 >> 172.21.15.38:0/3122243886 conn(0x7fa694102110 msgr2=0x7fa694104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 -- 172.21.15.38:0/3122243886 shutdown_connections 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.011+0000 7fa699f74700 1 -- 172.21.15.38:0/3122243886 wait complete. 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa699f74700 1 Processor -- start 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa699f74700 1 -- start start 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa699f74700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa699f74700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa69410c090 con 0x7fa694108740 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69419bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52534/0 (socket says 172.21.15.38:52534) 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa6937fe700 1 -- 172.21.15.38:0/350609648 learned_addr learned my addr 172.21.15.38:0/350609648 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa6937fe700 1 -- 172.21.15.38:0/350609648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69419c140 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.012+0000 7fa6937fe700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69419bc00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa684009530 tx=0x7fa68400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.013+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa684020020 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.013+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa684003b50 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.013+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69419c3d0 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.013+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa684017040 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.013+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa69419f0c0 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.013+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa684023020 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.014+0000 7fa690ff9700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fa680039050 0x7fa68003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.014+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa694063890 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.014+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa68400f050 con 0x7fa694108740 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.014+0000 7fa692ffd700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fa680039050 0x7fa68003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:02.030 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.014+0000 7fa692ffd700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fa680039050 0x7fa68003b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa67c006fd0 tx=0x7fa67c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.019+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa68402a020 con 0x7fa694108740 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.246+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fa694103330 con 0x7fa680039050 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.256+0000 7fa690ff9700 1 -- 172.21.15.38:0/350609648 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa694103330 con 0x7fa680039050 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fa680039050 msgr2=0x7fa68003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fa680039050 0x7fa68003b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa67c006fd0 tx=0x7fa67c006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 msgr2=0x7fa69419bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69419bc00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa684009530 tx=0x7fa68400bd50 comp rx=0 tx=0).stop 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 shutdown_connections 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fa680039050 0x7fa68003b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 --2- 172.21.15.38:0/350609648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa694108740 0x7fa69419bc00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 >> 172.21.15.38:0/350609648 conn(0x7fa694102110 msgr2=0x7fa694102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 shutdown_connections 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:01.258+0000 7fa699f74700 1 -- 172.21.15.38:0/350609648 wait complete. 2024-04-22T01:03:02.031 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mgr service... 2024-04-22T01:03:02.564 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:02 smithi038 ceph-mon[28043]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:02.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:02 smithi038 ceph-mon[28043]: Saving service mon spec with placement count:5 2024-04-22T01:03:02.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:02 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e2f7a700 1 Processor -- start 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e2f7a700 1 -- start start 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e2f7a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e2f7a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7dc1086d0 con 0x7fc7dc107ce0 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e0d16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e0d16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52540/0 (socket says 172.21.15.38:52540) 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.484+0000 7fc7e0d16700 1 -- 172.21.15.38:0/3919873861 learned_addr learned my addr 172.21.15.38:0/3919873861 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.485+0000 7fc7e0d16700 1 -- 172.21.15.38:0/3919873861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc108e40 con 0x7fc7dc107ce0 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.485+0000 7fc7e0d16700 1 --2- 172.21.15.38:0/3919873861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc108100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00098a0 tx=0x7fc7d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec11f9af7013e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.485+0000 7fc7db7fe700 1 -- 172.21.15.38:0/3919873861 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7d0004160 con 0x7fc7dc107ce0 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.485+0000 7fc7db7fe700 1 -- 172.21.15.38:0/3919873861 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc7d0004300 con 0x7fc7dc107ce0 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.485+0000 7fc7db7fe700 1 -- 172.21.15.38:0/3919873861 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7d0011b50 con 0x7fc7dc107ce0 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/3919873861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 msgr2=0x7fc7dc108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 --2- 172.21.15.38:0/3919873861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc108100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00098a0 tx=0x7fc7d0009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/3919873861 shutdown_connections 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 --2- 172.21.15.38:0/3919873861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc108100 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/3919873861 >> 172.21.15.38:0/3919873861 conn(0x7fc7dc103300 msgr2=0x7fc7dc105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/3919873861 shutdown_connections 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/3919873861 wait complete. 2024-04-22T01:03:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.486+0000 7fc7e2f7a700 1 Processor -- start 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e2f7a700 1 -- start start 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e2f7a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc19bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e2f7a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7dc10c1f0 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e0d16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc19bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e0d16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc19bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52552/0 (socket says 172.21.15.38:52552) 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e0d16700 1 -- 172.21.15.38:0/1320935967 learned_addr learned my addr 172.21.15.38:0/1320935967 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e0d16700 1 -- 172.21.15.38:0/1320935967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc19c170 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e0d16700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc19bc30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0011670 tx=0x7fc7d0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7d00039c0 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc7d0011e50 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.487+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7dc19c400 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.488+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7d001a5c0 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.488+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7dc19f0f0 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.489+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc7d001a870 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.489+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7dc063890 con 0x7fc7dc107ce0 2024-04-22T01:03:03.371 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.489+0000 7fc7d9ffb700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc7cc039050 0x7fc7cc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.489+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc7d004d5a0 con 0x7fc7dc107ce0 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.489+0000 7fc7dbfff700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc7cc039050 0x7fc7cc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.490+0000 7fc7dbfff700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc7cc039050 0x7fc7cc03b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8006fd0 tx=0x7fc7c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.494+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc7d0019c70 con 0x7fc7dc107ce0 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.719+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc7dc104540 con 0x7fc7cc039050 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.724+0000 7fc7d9ffb700 1 -- 172.21.15.38:0/1320935967 <== mgr.14120 v2:172.21.15.38:6800/3063830265 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc7dc104540 con 0x7fc7cc039050 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc7cc039050 msgr2=0x7fc7cc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc7cc039050 0x7fc7cc03b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8006fd0 tx=0x7fc7c8006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 msgr2=0x7fc7dc19bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc19bc30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0011670 tx=0x7fc7d0011b60 comp rx=0 tx=0).stop 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 shutdown_connections 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fc7cc039050 0x7fc7cc03b510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 --2- 172.21.15.38:0/1320935967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7dc107ce0 0x7fc7dc19bc30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 >> 172.21.15.38:0/1320935967 conn(0x7fc7dc103300 msgr2=0x7fc7dc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:03.372 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 shutdown_connections 2024-04-22T01:03:03.373 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:02.726+0000 7fc7e2f7a700 1 -- 172.21.15.38:0/1320935967 wait complete. 2024-04-22T01:03:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:03 smithi038 ceph-mon[28043]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:03 smithi038 ceph-mon[28043]: Saving service mgr spec with placement count:2 2024-04-22T01:03:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:03 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:04.810 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.787+0000 7fac3209a700 1 Processor -- start 2024-04-22T01:03:04.810 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.787+0000 7fac3209a700 1 -- start start 2024-04-22T01:03:04.810 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac3209a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:04.810 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac3209a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c07bdf0 con 0x7fac2c108750 2024-04-22T01:03:04.810 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c10ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52554/0 (socket says 172.21.15.38:52554) 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2b7fe700 1 -- 172.21.15.38:0/3768638668 learned_addr learned my addr 172.21.15.38:0/3768638668 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2b7fe700 1 -- 172.21.15.38:0/3768638668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c10b080 con 0x7fac2c108750 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2b7fe700 1 --2- 172.21.15.38:0/3768638668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c10ab40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0098a0 tx=0x7fac1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61882e7961270643 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2a7fc700 1 -- 172.21.15.38:0/3768638668 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c004160 con 0x7fac2c108750 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2a7fc700 1 -- 172.21.15.38:0/3768638668 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac1c004300 con 0x7fac2c108750 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.788+0000 7fac2a7fc700 1 -- 172.21.15.38:0/3768638668 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c011b50 con 0x7fac2c108750 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 -- 172.21.15.38:0/3768638668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 msgr2=0x7fac2c10ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 --2- 172.21.15.38:0/3768638668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c10ab40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0098a0 tx=0x7fac1c009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 -- 172.21.15.38:0/3768638668 shutdown_connections 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 --2- 172.21.15.38:0/3768638668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c10ab40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 -- 172.21.15.38:0/3768638668 >> 172.21.15.38:0/3768638668 conn(0x7fac2c102160 msgr2=0x7fac2c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 -- 172.21.15.38:0/3768638668 shutdown_connections 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.789+0000 7fac3209a700 1 -- 172.21.15.38:0/3768638668 wait complete. 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac3209a700 1 Processor -- start 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac3209a700 1 -- start start 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac3209a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac3209a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c10c0a0 con 0x7fac2c108750 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac2b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac2b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c19bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52556/0 (socket says 172.21.15.38:52556) 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.790+0000 7fac2b7fe700 1 -- 172.21.15.38:0/1511796055 learned_addr learned my addr 172.21.15.38:0/1511796055 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac2b7fe700 1 -- 172.21.15.38:0/1511796055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c19c1b0 con 0x7fac2c108750 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac2b7fe700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c19bc70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fb0 tx=0x7fac1c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c0038b0 con 0x7fac2c108750 2024-04-22T01:03:04.812 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac1c011e50 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2c19c440 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c019420 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.791+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2c19f130 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.792+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fac1c019740 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.792+0000 7fac28ff9700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fac14039000 0x7fac1403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.792+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fac1c04d350 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.792+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac2c063890 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.792+0000 7fac2affd700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fac14039000 0x7fac1403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.793+0000 7fac2affd700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fac14039000 0x7fac1403b4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac20006fd0 tx=0x7fac20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:03.797+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fac1c00bc70 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.005+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fac2c19f6d0 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.007+0000 7fac28ff9700 1 -- 172.21.15.38:0/1511796055 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7fac1c00fda0 con 0x7fac2c108750 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fac14039000 msgr2=0x7fac1403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:04.813 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fac14039000 0x7fac1403b4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac20006fd0 tx=0x7fac20006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 msgr2=0x7fac2c19bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c19bc70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fb0 tx=0x7fac1c011bf0 comp rx=0 tx=0).stop 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 shutdown_connections 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fac14039000 0x7fac1403b4c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 --2- 172.21.15.38:0/1511796055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c108750 0x7fac2c19bc70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 >> 172.21.15.38:0/1511796055 conn(0x7fac2c102160 msgr2=0x7fac2c102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 shutdown_connections 2024-04-22T01:03:04.814 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:04.011+0000 7fac3209a700 1 -- 172.21.15.38:0/1511796055 wait complete. 2024-04-22T01:03:05.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1511796055' entity='client.admin' 2024-04-22T01:03:05.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:05 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:05.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:05 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:06.854 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3023140624' entity='client.admin' 2024-04-22T01:03:06.854 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:06 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.311+0000 7fe8e4f68700 1 Processor -- start 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.311+0000 7fe8e4f68700 1 -- start start 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.311+0000 7fe8e4f68700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e01063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.311+0000 7fe8e4f68700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e0106970 con 0x7fe8e0105f80 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e01063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e01063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52562/0 (socket says 172.21.15.38:52562) 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8de59c700 1 -- 172.21.15.38:0/716555584 learned_addr learned my addr 172.21.15.38:0/716555584 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8de59c700 1 -- 172.21.15.38:0/716555584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e0108f10 con 0x7fe8e0105f80 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8de59c700 1 --2- 172.21.15.38:0/716555584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e01063a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40098a0 tx=0x7fe8d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b0ca111283d209 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:06.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8dd59a700 1 -- 172.21.15.38:0/716555584 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d4004160 con 0x7fe8e0105f80 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8dd59a700 1 -- 172.21.15.38:0/716555584 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8d4004300 con 0x7fe8e0105f80 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.312+0000 7fe8dd59a700 1 -- 172.21.15.38:0/716555584 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d4011b20 con 0x7fe8e0105f80 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 -- 172.21.15.38:0/716555584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 msgr2=0x7fe8e01063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 --2- 172.21.15.38:0/716555584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e01063a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40098a0 tx=0x7fe8d4009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 -- 172.21.15.38:0/716555584 shutdown_connections 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 --2- 172.21.15.38:0/716555584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e01063a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 -- 172.21.15.38:0/716555584 >> 172.21.15.38:0/716555584 conn(0x7fe8e01018f0 msgr2=0x7fe8e0103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 -- 172.21.15.38:0/716555584 shutdown_connections 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.313+0000 7fe8e4f68700 1 -- 172.21.15.38:0/716555584 wait complete. 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8e4f68700 1 Processor -- start 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8e4f68700 1 -- start start 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8e4f68700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8e4f68700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e010c2c0 con 0x7fe8e0105f80 2024-04-22T01:03:06.863 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8de59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52568/0 (socket says 172.21.15.38:52568) 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8de59c700 1 -- 172.21.15.38:0/3023140624 learned_addr learned my addr 172.21.15.38:0/3023140624 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.314+0000 7fe8de59c700 1 -- 172.21.15.38:0/3023140624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e019c170 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.315+0000 7fe8de59c700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e019bc30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40116a0 tx=0x7fe8d4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.315+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d4003a40 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.315+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8d4003be0 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.315+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e019c400 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.315+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8d401a720 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.315+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e019c860 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.316+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe8d401aa20 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.316+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e0063890 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.316+0000 7fe8cf7fe700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fe8c8039000 0x7fe8c803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.316+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe8d404d6c0 con 0x7fe8e0105f80 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.316+0000 7fe8ddd9b700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fe8c8039000 0x7fe8c803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:06.864 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.317+0000 7fe8ddd9b700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fe8c8039000 0x7fe8c803b4c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006fd0 tx=0x7fe8d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.321+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe8d4011e20 con 0x7fe8e0105f80 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.545+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fe8e019f2d0 con 0x7fe8e0105f80 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.547+0000 7fe8cf7fe700 1 -- 172.21.15.38:0/3023140624 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fe8d401adc0 con 0x7fe8e0105f80 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fe8c8039000 msgr2=0x7fe8c803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fe8c8039000 0x7fe8c803b4c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006fd0 tx=0x7fe8d0006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 msgr2=0x7fe8e019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e019bc30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40116a0 tx=0x7fe8d4011ba0 comp rx=0 tx=0).stop 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 shutdown_connections 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fe8c8039000 0x7fe8c803b4c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 --2- 172.21.15.38:0/3023140624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8e0105f80 0x7fe8e019bc30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.550+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 >> 172.21.15.38:0/3023140624 conn(0x7fe8e01018f0 msgr2=0x7fe8e0102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.551+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 shutdown_connections 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:05.551+0000 7fe8e4f68700 1 -- 172.21.15.38:0/3023140624 wait complete. 2024-04-22T01:03:06.865 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2024-04-22T01:03:08.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:08 smithi038 ceph-mon[28043]: from='mgr.14120 172.21.15.38:0/3202231649' entity='mgr.x' 2024-04-22T01:03:08.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:08 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1576630101' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T01:03:08.905 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: ignoring --setuser ceph since I am not root 2024-04-22T01:03:08.905 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: ignoring --setgroup ceph since I am not root 2024-04-22T01:03:08.905 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:08.758+0000 7fc4bf0b3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:03:09.348 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:08.948+0000 7fc4bf0b3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:03:09.348 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:09.056+0000 7fc4bf0b3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:03:09.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.314+0000 7fd391200700 1 Processor -- start 2024-04-22T01:03:09.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.314+0000 7fd391200700 1 -- start start 2024-04-22T01:03:09.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.315+0000 7fd391200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c10ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.315+0000 7fd391200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c07bdd0 con 0x7fd38c108770 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.315+0000 7fd38ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c10ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.315+0000 7fd38ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c10ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56126/0 (socket says 172.21.15.38:56126) 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.315+0000 7fd38ad9d700 1 -- 172.21.15.38:0/199979508 learned_addr learned my addr 172.21.15.38:0/199979508 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.315+0000 7fd38ad9d700 1 -- 172.21.15.38:0/199979508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c10b0a0 con 0x7fd38c108770 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.316+0000 7fd38ad9d700 1 --2- 172.21.15.38:0/199979508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c10ab60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0098a0 tx=0x7fd37c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf06d406f6b6cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.316+0000 7fd389d9b700 1 -- 172.21.15.38:0/199979508 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd37c004160 con 0x7fd38c108770 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.316+0000 7fd389d9b700 1 -- 172.21.15.38:0/199979508 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd37c004300 con 0x7fd38c108770 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.316+0000 7fd389d9b700 1 -- 172.21.15.38:0/199979508 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd37c011b20 con 0x7fd38c108770 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 -- 172.21.15.38:0/199979508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 msgr2=0x7fd38c10ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 --2- 172.21.15.38:0/199979508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c10ab60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0098a0 tx=0x7fd37c009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 -- 172.21.15.38:0/199979508 shutdown_connections 2024-04-22T01:03:09.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 --2- 172.21.15.38:0/199979508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c10ab60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 -- 172.21.15.38:0/199979508 >> 172.21.15.38:0/199979508 conn(0x7fd38c102160 msgr2=0x7fd38c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 -- 172.21.15.38:0/199979508 shutdown_connections 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.317+0000 7fd391200700 1 -- 172.21.15.38:0/199979508 wait complete. 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.318+0000 7fd391200700 1 Processor -- start 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.318+0000 7fd391200700 1 -- start start 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.318+0000 7fd391200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c19bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.318+0000 7fd391200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10c0b0 con 0x7fd38c108770 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd38ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c19bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd38ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c19bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56134/0 (socket says 172.21.15.38:56134) 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd38ad9d700 1 -- 172.21.15.38:0/1576630101 learned_addr learned my addr 172.21.15.38:0/1576630101 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd38ad9d700 1 -- 172.21.15.38:0/1576630101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c19c0f0 con 0x7fd38c108770 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd38ad9d700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c19bbb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0117e0 tx=0x7fd37c003980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd37c011b20 con 0x7fd38c108770 2024-04-22T01:03:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd37c003c50 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c19c380 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.319+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c19c7e0 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.320+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd37c024870 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.320+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd37c02d3f0 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.321+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c063890 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.321+0000 7fd373fff700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd374039050 0x7fd37403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.321+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd37c04d420 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.321+0000 7fd38a59c700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd374039050 0x7fd37403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.321+0000 7fd38a59c700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd374039050 0x7fd37403b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd380006fd0 tx=0x7fd380006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.327+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd37c011e20 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:07.619+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fd38c19f270 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.570+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fd37c00b7e0 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.572+0000 7fd373fff700 1 -- 172.21.15.38:0/1576630101 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fd37c01ac90 con 0x7fd38c108770 2024-04-22T01:03:09.359 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd374039050 msgr2=0x7fd37403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd374039050 0x7fd37403b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd380006fd0 tx=0x7fd380006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 msgr2=0x7fd38c19bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c19bbb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0117e0 tx=0x7fd37c003980 comp rx=0 tx=0).stop 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 shutdown_connections 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd374039050 0x7fd37403b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 --2- 172.21.15.38:0/1576630101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd38c108770 0x7fd38c19bbb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 >> 172.21.15.38:0/1576630101 conn(0x7fd38c102160 msgr2=0x7fd38c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 shutdown_connections 2024-04-22T01:03:09.360 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:08.573+0000 7fd391200700 1 -- 172.21.15.38:0/1576630101 wait complete. 2024-04-22T01:03:09.612 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:09 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1576630101' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T01:03:09.612 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:09 smithi038 ceph-mon[28043]: mgrmap e9: x(active, since 20s) 2024-04-22T01:03:09.612 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:09.380+0000 7fc4bf0b3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:03:09.612 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:09.470+0000 7fc4bf0b3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:03:10.864 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:10 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3029335784' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T01:03:10.865 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:10 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:10.731+0000 7fc4bf0b3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:03:11.154 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:10 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:10.910+0000 7fc4bf0b3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:03:11.155 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:11 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:11.087+0000 7fc4bf0b3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:03:11.181 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:03:11.181 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T01:03:11.181 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T01:03:11.181 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.247+0000 7f18c6d05700 1 Processor -- start 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.247+0000 7f18c6d05700 1 -- start start 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.248+0000 7f18c6d05700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c0106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.248+0000 7f18c6d05700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f18c0107580 con 0x7f18c0106b90 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.248+0000 7f18c4aa1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c0106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.248+0000 7f18c4aa1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c0106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56162/0 (socket says 172.21.15.38:56162) 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.248+0000 7f18c4aa1700 1 -- 172.21.15.38:0/2463397409 learned_addr learned my addr 172.21.15.38:0/2463397409 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.248+0000 7f18c4aa1700 1 -- 172.21.15.38:0/2463397409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c0107d90 con 0x7f18c0106b90 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.249+0000 7f18c4aa1700 1 --2- 172.21.15.38:0/2463397409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c0106fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098a0 tx=0x7f18b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39fb95aa808afb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.249+0000 7f18bf7fe700 1 -- 172.21.15.38:0/2463397409 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18b4004160 con 0x7f18c0106b90 2024-04-22T01:03:11.182 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.249+0000 7f18bf7fe700 1 -- 172.21.15.38:0/2463397409 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b4004300 con 0x7f18c0106b90 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.249+0000 7f18bf7fe700 1 -- 172.21.15.38:0/2463397409 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18b4011b20 con 0x7f18c0106b90 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 -- 172.21.15.38:0/2463397409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 msgr2=0x7f18c0106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 --2- 172.21.15.38:0/2463397409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c0106fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098a0 tx=0x7f18b4009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 -- 172.21.15.38:0/2463397409 shutdown_connections 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 --2- 172.21.15.38:0/2463397409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c0106fb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 -- 172.21.15.38:0/2463397409 >> 172.21.15.38:0/2463397409 conn(0x7f18c0102110 msgr2=0x7f18c0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 -- 172.21.15.38:0/2463397409 shutdown_connections 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 -- 172.21.15.38:0/2463397409 wait complete. 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 Processor -- start 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.250+0000 7f18c6d05700 1 -- start start 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c6d05700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c6d05700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f18c010b140 con 0x7f18c0106b90 2024-04-22T01:03:11.183 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c4aa1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c4aa1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c019bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56172/0 (socket says 172.21.15.38:56172) 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c4aa1700 1 -- 172.21.15.38:0/3029335784 learned_addr learned my addr 172.21.15.38:0/3029335784 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c4aa1700 1 -- 172.21.15.38:0/3029335784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c019c1a0 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c4aa1700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c019bc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18b4000c00 tx=0x7f18b4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18b4003b60 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b401a910 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c019c430 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.251+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18b4024930 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.252+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c019c890 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.253+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c0063890 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.253+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f18b4022020 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.253+0000 7f18bdffb700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 0x7f18b003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.253+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f18b404d3c0 con 0x7f18c0106b90 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.253+0000 7f18bffff700 1 -- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 msgr2=0x7f18b003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:11.184 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.253+0000 7f18bffff700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 0x7f18b003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.258+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f18b4005a20 con 0x7f18c0106b90 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.454+0000 7f18bffff700 1 -- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 msgr2=0x7f18b003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.454+0000 7f18bffff700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 0x7f18b003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.523+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f18c0062e40 con 0x7f18c0106b90 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.523+0000 7f18bdffb700 1 -- 172.21.15.38:0/3029335784 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f18b400bcc0 con 0x7f18c0106b90 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 msgr2=0x7f18b003b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 0x7f18b003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 msgr2=0x7f18c019bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c019bc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18b4000c00 tx=0x7f18b4011c10 comp rx=0 tx=0).stop 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 shutdown_connections 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7f18b0039100 0x7f18b003b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 --2- 172.21.15.38:0/3029335784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18c0106b90 0x7f18c019bc60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 >> 172.21.15.38:0/3029335784 conn(0x7f18c0102110 msgr2=0x7f18c0102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 shutdown_connections 2024-04-22T01:03:11.185 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:10.526+0000 7f18c6d05700 1 -- 172.21.15.38:0/3029335784 wait complete. 2024-04-22T01:03:11.186 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-04-22T01:03:11.186 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2024-04-22T01:03:11.437 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:11 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:11.392+0000 7fc4bf0b3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:03:11.706 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:11 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:11.492+0000 7fc4bf0b3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:03:11.706 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:11 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:11.706+0000 7fc4bf0b3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:03:12.155 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:11 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:11.906+0000 7fc4bf0b3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:03:12.904 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:12 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:12.534+0000 7fc4bf0b3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:03:13.281 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:12 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:12.972+0000 7fc4bf0b3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:03:13.281 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:13 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:13.072+0000 7fc4bf0b3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:03:13.282 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:13 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:13.280+0000 7fc4bf0b3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:03:14.154 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:13 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:13.824+0000 7fc4bf0b3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:03:14.155 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:13 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:13.920+0000 7fc4bf0b3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:03:14.530 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:14 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:14.172+0000 7fc4bf0b3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:03:14.530 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:14 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:14.311+0000 7fc4bf0b3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:03:14.530 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:14 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:14.410+0000 7fc4bf0b3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:03:14.531 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:14 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:14.530+0000 7fc4bf0b3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:03:14.905 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:14 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:14.625+0000 7fc4bf0b3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:03:15.746 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:15.470+0000 7fc4bf0b3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:03:15.746 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:03:15.693+0000 7fc4bf0b3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: Active manager daemon x restarted 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: Activating manager daemon x 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: mgrmap e10: x(active, starting, since 0.00593498s) 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T01:03:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T01:03:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: Manager daemon x is now available 2024-04-22T01:03:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: [22/Apr/2024:01:03:16] ENGINE Bus STARTING 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: [22/Apr/2024:01:03:16] ENGINE Serving on http://172.21.15.38:8765 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: [22/Apr/2024:01:03:16] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: [22/Apr/2024:01:03:16] ENGINE Bus STARTED 2024-04-22T01:03:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:16 smithi038 ceph-mon[28043]: mgrmap e11: x(active, since 1.00645s) 2024-04-22T01:03:17.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-22T01:03:17.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.641+0000 7fd6fe7fe700 1 Processor -- start 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.641+0000 7fd6fe7fe700 1 -- start start 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6fe7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f81063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6fe7fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f8106970 con 0x7fd6f8105f80 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f81063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f81063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56176/0 (socket says 172.21.15.38:56176) 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f7fff700 1 -- 172.21.15.38:0/875401774 learned_addr learned my addr 172.21.15.38:0/875401774 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f7fff700 1 -- 172.21.15.38:0/875401774 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f8108fb0 con 0x7fd6f8105f80 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f7fff700 1 --2- 172.21.15.38:0/875401774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f81063a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098a0 tx=0x7fd6e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0495effc0de943 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:17.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f6ffd700 1 -- 172.21.15.38:0/875401774 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0004160 con 0x7fd6f8105f80 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f6ffd700 1 -- 172.21.15.38:0/875401774 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e0004300 con 0x7fd6f8105f80 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.642+0000 7fd6f6ffd700 1 -- 172.21.15.38:0/875401774 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0011b20 con 0x7fd6f8105f80 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/875401774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 msgr2=0x7fd6f81063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/875401774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f81063a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098a0 tx=0x7fd6e0009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/875401774 shutdown_connections 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/875401774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f81063a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/875401774 >> 172.21.15.38:0/875401774 conn(0x7fd6f81018f0 msgr2=0x7fd6f8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/875401774 shutdown_connections 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.643+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/875401774 wait complete. 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6fe7fe700 1 Processor -- start 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6fe7fe700 1 -- start start 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6fe7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6fe7fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f810c1e0 con 0x7fd6f8105f80 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:17.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f819bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56188/0 (socket says 172.21.15.38:56188) 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.644+0000 7fd6f7fff700 1 -- 172.21.15.38:0/2109179024 learned_addr learned my addr 172.21.15.38:0/2109179024 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6f7fff700 1 -- 172.21.15.38:0/2109179024 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f819c170 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6f7fff700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f819bc30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00116a0 tx=0x7fd6e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0003a40 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f819c400 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e0003be0 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e001a720 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.645+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f819c860 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.646+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fd6e001aa20 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.646+0000 7fd6f57fa700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.646+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 --> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd6e403bc20 con 0x7fd6e4039050 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.646+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd6e0024aa0 con 0x7fd6f8105f80 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.646+0000 7fd6f77fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 msgr2=0x7fd6e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.646+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:03:17.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.847+0000 7fd6f77fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 msgr2=0x7fd6e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:11.847+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:12.248+0000 7fd6f77fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 msgr2=0x7fd6e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:12.248+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:13.049+0000 7fd6f77fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 msgr2=0x7fd6e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:13.049+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:14.650+0000 7fd6f77fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 msgr2=0x7fd6e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3063830265 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:14.650+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:15.699+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7fd6e002d910 con 0x7fd6f8105f80 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:15.699+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 msgr2=0x7fd6e403b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:15.699+0000 7fd6f57fa700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.698+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fd6e004f340 con 0x7fd6f8105f80 2024-04-22T01:03:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.699+0000 7fd6f57fa700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6e403cb10 0x7fd6e403ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.699+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd6e403bc20 con 0x7fd6e403cb10 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.699+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6e403cb10 0x7fd6e403ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.700+0000 7fd6f77fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6e403cb10 0x7fd6e403ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8003a10 tx=0x7fd6e80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.700+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd6e403bc20 con 0x7fd6e403cb10 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.707+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd6f810c360 con 0x7fd6e403cb10 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.707+0000 7fd6f57fa700 1 -- 172.21.15.38:0/2109179024 <== mgr.14152 v2:172.21.15.38:6800/3086698001 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fd6f810c360 con 0x7fd6e403cb10 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6e403cb10 msgr2=0x7fd6e403ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6e403cb10 0x7fd6e403ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8003a10 tx=0x7fd6e80092b0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 msgr2=0x7fd6f819bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f819bc30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00116a0 tx=0x7fd6e0011ba0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 shutdown_connections 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6e403cb10 0x7fd6e403ef00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f8105f80 0x7fd6f819bc30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 --2- 172.21.15.38:0/2109179024 >> [v2:172.21.15.38:6800/3063830265,v1:172.21.15.38:6801/3063830265] conn(0x7fd6e4039050 0x7fd6e403b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 >> 172.21.15.38:0/2109179024 conn(0x7fd6f81018f0 msgr2=0x7fd6f81023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 shutdown_connections 2024-04-22T01:03:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:16.708+0000 7fd6fe7fe700 1 -- 172.21.15.38:0/2109179024 wait complete. 2024-04-22T01:03:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2024-04-22T01:03:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2024-04-22T01:03:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:18 smithi038 ceph-mon[28043]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T01:03:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:18 smithi038 ceph-mon[28043]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T01:03:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:18.968 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T01:03:18.968 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.004+0000 7fc93379b700 1 Processor -- start 2024-04-22T01:03:18.968 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc93379b700 1 -- start start 2024-04-22T01:03:18.968 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc93379b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:18.968 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc93379b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c07bdf0 con 0x7fc92c108750 2024-04-22T01:03:18.968 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc931537700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:18.969 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc931537700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c10ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59412/0 (socket says 172.21.15.38:59412) 2024-04-22T01:03:18.969 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc931537700 1 -- 172.21.15.38:0/4070728504 learned_addr learned my addr 172.21.15.38:0/4070728504 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:18.969 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.005+0000 7fc931537700 1 -- 172.21.15.38:0/4070728504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c10b080 con 0x7fc92c108750 2024-04-22T01:03:18.969 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc931537700 1 --2- 172.21.15.38:0/4070728504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c10ab40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0098a0 tx=0x7fc91c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de6073e9a341801 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:18.969 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc923fff700 1 -- 172.21.15.38:0/4070728504 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc91c004160 con 0x7fc92c108750 2024-04-22T01:03:18.969 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc923fff700 1 -- 172.21.15.38:0/4070728504 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc91c004300 con 0x7fc92c108750 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc923fff700 1 -- 172.21.15.38:0/4070728504 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc91c011b20 con 0x7fc92c108750 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc93379b700 1 -- 172.21.15.38:0/4070728504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 msgr2=0x7fc92c10ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc93379b700 1 --2- 172.21.15.38:0/4070728504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c10ab40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0098a0 tx=0x7fc91c009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc93379b700 1 -- 172.21.15.38:0/4070728504 shutdown_connections 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.006+0000 7fc93379b700 1 --2- 172.21.15.38:0/4070728504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c10ab40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 -- 172.21.15.38:0/4070728504 >> 172.21.15.38:0/4070728504 conn(0x7fc92c102160 msgr2=0x7fc92c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 -- 172.21.15.38:0/4070728504 shutdown_connections 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 -- 172.21.15.38:0/4070728504 wait complete. 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 Processor -- start 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 -- start start 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c107fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:18.970 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.007+0000 7fc93379b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10c0e0 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc931537700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c107fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc931537700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c107fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59414/0 (socket says 172.21.15.38:59414) 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc931537700 1 -- 172.21.15.38:0/1010633361 learned_addr learned my addr 172.21.15.38:0/1010633361 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc931537700 1 -- 172.21.15.38:0/1010633361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c108510 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc931537700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c107fd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0116a0 tx=0x7fc91c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc91c003a40 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc91c003be0 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c106700 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc91c01a750 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.008+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c106bc0 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.009+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fc91c024430 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.009+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc92c063890 con 0x7fc92c108750 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.009+0000 7fc9227fc700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc918038f40 0x7fc91803b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:18.971 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.009+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc91c04d690 con 0x7fc92c108750 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.010+0000 7fc930d36700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc918038f40 0x7fc91803b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.010+0000 7fc930d36700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc918038f40 0x7fc91803b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc928006fd0 tx=0x7fc928006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.015+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc91c011e20 con 0x7fc92c108750 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.231+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fc92c1074d0 con 0x7fc918038f40 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.320+0000 7fc9227fc700 1 -- 172.21.15.38:0/1010633361 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fc92c1074d0 con 0x7fc918038f40 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc918038f40 msgr2=0x7fc91803b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc918038f40 0x7fc91803b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc928006fd0 tx=0x7fc928006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 msgr2=0x7fc92c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c107fd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0116a0 tx=0x7fc91c011ba0 comp rx=0 tx=0).stop 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 shutdown_connections 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc918038f40 0x7fc91803b400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 --2- 172.21.15.38:0/1010633361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc92c108750 0x7fc92c107fd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:18.972 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.322+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 >> 172.21.15.38:0/1010633361 conn(0x7fc92c102160 msgr2=0x7fc92c102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:18.973 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.323+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 shutdown_connections 2024-04-22T01:03:18.973 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:18.323+0000 7fc93379b700 1 -- 172.21.15.38:0/1010633361 wait complete. 2024-04-22T01:03:18.973 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2024-04-22T01:03:19.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:19 smithi038 ceph-mon[28043]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:19.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:19 smithi038 ceph-mon[28043]: mgrmap e12: x(active, since 2s) 2024-04-22T01:03:19.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:20.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ZfD6gIflnQaPWh9kyNdsYutgoWbNcdyF3MSkttIInEt18HQHnlyTW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713747799, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T01:03:20.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f45062b3700 1 Processor -- start 2024-04-22T01:03:20.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f45062b3700 1 -- start start 2024-04-22T01:03:20.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f45062b3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f45001080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:20.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f45062b3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4500108690 con 0x7f4500107ca0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f44fffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f45001080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f44fffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f45001080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59422/0 (socket says 172.21.15.38:59422) 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.401+0000 7f44fffff700 1 -- 172.21.15.38:0/3858460029 learned_addr learned my addr 172.21.15.38:0/3858460029 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.402+0000 7f44fffff700 1 -- 172.21.15.38:0/3858460029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4500108e50 con 0x7f4500107ca0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.402+0000 7f44fffff700 1 --2- 172.21.15.38:0/3858460029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f45001080c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44f00098a0 tx=0x7f44f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f7a65f2bbc7ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.402+0000 7f44feffd700 1 -- 172.21.15.38:0/3858460029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44f0004160 con 0x7f4500107ca0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.402+0000 7f44feffd700 1 -- 172.21.15.38:0/3858460029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f0004300 con 0x7f4500107ca0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.402+0000 7f44feffd700 1 -- 172.21.15.38:0/3858460029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44f0011b20 con 0x7f4500107ca0 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 -- 172.21.15.38:0/3858460029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 msgr2=0x7f45001080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 --2- 172.21.15.38:0/3858460029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f45001080c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44f00098a0 tx=0x7f44f0009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 -- 172.21.15.38:0/3858460029 shutdown_connections 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 --2- 172.21.15.38:0/3858460029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f45001080c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 -- 172.21.15.38:0/3858460029 >> 172.21.15.38:0/3858460029 conn(0x7f45001032a0 msgr2=0x7f45001056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 -- 172.21.15.38:0/3858460029 shutdown_connections 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 -- 172.21.15.38:0/3858460029 wait complete. 2024-04-22T01:03:20.447 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 Processor -- start 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.403+0000 7f45062b3700 1 -- start start 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f45062b3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f450007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f45062b3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f450010c200 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f450007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f450007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59432/0 (socket says 172.21.15.38:59432) 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fffff700 1 -- 172.21.15.38:0/214673191 learned_addr learned my addr 172.21.15.38:0/214673191 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fffff700 1 -- 172.21.15.38:0/214673191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450007e030 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fffff700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f450007daf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f44f00116a0 tx=0x7f44f0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44f0003a40 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f0003be0 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450007c220 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.404+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44f001a720 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.405+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450007c6e0 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.405+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f44f001aa20 con 0x7f4500107ca0 2024-04-22T01:03:20.448 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.406+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4500063890 con 0x7f4500107ca0 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.406+0000 7f44fd7fa700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44e8039050 0x7f44e803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.406+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f44f004d7e0 con 0x7f4500107ca0 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.406+0000 7f44ff7fe700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44e8039050 0x7f44e803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.406+0000 7f44ff7fe700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44e8039050 0x7f44e803b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f44f4006fd0 tx=0x7f44f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.411+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f44f0011e20 con 0x7f4500107ca0 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.639+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f450007c960 con 0x7f44e8039050 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.888+0000 7f44fd7fa700 1 -- 172.21.15.38:0/214673191 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f450007c960 con 0x7f44e8039050 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44e8039050 msgr2=0x7f44e803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44e8039050 0x7f44e803b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f44f4006fd0 tx=0x7f44f4006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 msgr2=0x7f450007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f450007daf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f44f00116a0 tx=0x7f44f0011ba0 comp rx=0 tx=0).stop 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 shutdown_connections 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44e8039050 0x7f44e803b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:20.449 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 --2- 172.21.15.38:0/214673191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4500107ca0 0x7f450007daf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:20.450 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 >> 172.21.15.38:0/214673191 conn(0x7f45001032a0 msgr2=0x7f4500103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:20.450 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 shutdown_connections 2024-04-22T01:03:20.450 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:19.890+0000 7f45062b3700 1 -- 172.21.15.38:0/214673191 wait complete. 2024-04-22T01:03:20.450 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2024-04-22T01:03:21.121 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:20 smithi038 ceph-mon[28043]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:21.121 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:22.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:21 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2873135557' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc48331700 1 Processor -- start 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc48331700 1 -- start start 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc48331700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc40106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc48331700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc40107580 con 0x7fbc40106b90 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc460cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc40106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc460cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc40106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59442/0 (socket says 172.21.15.38:59442) 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.870+0000 7fbc460cd700 1 -- 172.21.15.38:0/2710628947 learned_addr learned my addr 172.21.15.38:0/2710628947 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:22.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.871+0000 7fbc460cd700 1 -- 172.21.15.38:0/2710628947 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc40107d90 con 0x7fbc40106b90 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.871+0000 7fbc460cd700 1 --2- 172.21.15.38:0/2710628947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc40106fb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009e80 tx=0x7fbc3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874cb24fdb6c8fce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.871+0000 7fbc450cb700 1 -- 172.21.15.38:0/2710628947 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc30017070 con 0x7fbc40106b90 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.871+0000 7fbc450cb700 1 -- 172.21.15.38:0/2710628947 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc3000b8c0 con 0x7fbc40106b90 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.871+0000 7fbc450cb700 1 -- 172.21.15.38:0/2710628947 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc300045c0 con 0x7fbc40106b90 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 -- 172.21.15.38:0/2710628947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 msgr2=0x7fbc40106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 --2- 172.21.15.38:0/2710628947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc40106fb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009e80 tx=0x7fbc3000b330 comp rx=0 tx=0).stop 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 -- 172.21.15.38:0/2710628947 shutdown_connections 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 --2- 172.21.15.38:0/2710628947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc40106fb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 -- 172.21.15.38:0/2710628947 >> 172.21.15.38:0/2710628947 conn(0x7fbc40102110 msgr2=0x7fbc40104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 -- 172.21.15.38:0/2710628947 shutdown_connections 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.872+0000 7fbc48331700 1 -- 172.21.15.38:0/2710628947 wait complete. 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc48331700 1 Processor -- start 2024-04-22T01:03:22.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc48331700 1 -- start start 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc48331700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc4007b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc48331700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4010b140 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc460cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc4007b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc460cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc4007b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59458/0 (socket says 172.21.15.38:59458) 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc460cd700 1 -- 172.21.15.38:0/2873135557 learned_addr learned my addr 172.21.15.38:0/2873135557 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.873+0000 7fbc460cd700 1 -- 172.21.15.38:0/2873135557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4019e990 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.874+0000 7fbc460cd700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc4007b9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbc30003fa0 tx=0x7fbc30004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.874+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc30020020 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.874+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc3000bba0 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.874+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4019eb60 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.874+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc30017040 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.874+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4019ef80 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.875+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fbc30023020 con 0x7fbc40106b90 2024-04-22T01:03:22.250 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.875+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc40063890 con 0x7fbc40106b90 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.875+0000 7fbc3f7fe700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbc280390b0 0x7fbc2803b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.875+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc3000f050 con 0x7fbc40106b90 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.875+0000 7fbc458cc700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbc280390b0 0x7fbc2803b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.876+0000 7fbc458cc700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbc280390b0 0x7fbc2803b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbc34006fd0 tx=0x7fbc34006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:20.880+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbc3002a020 con 0x7fbc40106b90 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.107+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fbc4019f690 con 0x7fbc40106b90 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.108+0000 7fbc3f7fe700 1 -- 172.21.15.38:0/2873135557 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fbc3002d350 con 0x7fbc40106b90 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbc280390b0 msgr2=0x7fbc2803b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbc280390b0 0x7fbc2803b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbc34006fd0 tx=0x7fbc34006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 msgr2=0x7fbc4007b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc4007b9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbc30003fa0 tx=0x7fbc30004080 comp rx=0 tx=0).stop 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 shutdown_connections 2024-04-22T01:03:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbc280390b0 0x7fbc2803b570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:22.252 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 --2- 172.21.15.38:0/2873135557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc40106b90 0x7fbc4007b9f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:22.252 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 >> 172.21.15.38:0/2873135557 conn(0x7fbc40102110 msgr2=0x7fbc40102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:22.252 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 shutdown_connections 2024-04-22T01:03:22.252 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:21.110+0000 7fbc48331700 1 -- 172.21.15.38:0/2873135557 wait complete. 2024-04-22T01:03:22.257 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T01:03:22.257 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-22T01:03:22.266 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T01:03:22.266 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-22T01:03:22.266 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-22T01:03:22.266 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038:8443/ 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: Password: 1ilk5jhr3z 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:22.268 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config directory 2024-04-22T01:03:22.269 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2024-04-22T01:03:22.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:22 smithi038 ceph-mon[28043]: mgrmap e13: x(active, since 6s) 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb7088700 1 Processor -- start 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb7088700 1 -- start start 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb7088700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb01080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb7088700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb0108690 con 0x7f8fb0107ca0 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb4e24700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb01080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb4e24700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb01080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59474/0 (socket says 172.21.15.38:59474) 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.157+0000 7f8fb4e24700 1 -- 172.21.15.38:0/3639183121 learned_addr learned my addr 172.21.15.38:0/3639183121 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:23.940 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.158+0000 7f8fb4e24700 1 -- 172.21.15.38:0/3639183121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb0108e50 con 0x7f8fb0107ca0 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.158+0000 7f8fb4e24700 1 --2- 172.21.15.38:0/3639183121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb01080c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40098a0 tx=0x7f8fa4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752e158f745c6a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.158+0000 7f8faf7fe700 1 -- 172.21.15.38:0/3639183121 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fa4004160 con 0x7f8fb0107ca0 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.158+0000 7f8faf7fe700 1 -- 172.21.15.38:0/3639183121 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004300 con 0x7f8fb0107ca0 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.158+0000 7f8faf7fe700 1 -- 172.21.15.38:0/3639183121 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fa4011b20 con 0x7f8fb0107ca0 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 -- 172.21.15.38:0/3639183121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 msgr2=0x7f8fb01080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 --2- 172.21.15.38:0/3639183121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb01080c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40098a0 tx=0x7f8fa4009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 -- 172.21.15.38:0/3639183121 shutdown_connections 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 --2- 172.21.15.38:0/3639183121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb01080c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 -- 172.21.15.38:0/3639183121 >> 172.21.15.38:0/3639183121 conn(0x7f8fb01032a0 msgr2=0x7f8fb01056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 -- 172.21.15.38:0/3639183121 shutdown_connections 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 -- 172.21.15.38:0/3639183121 wait complete. 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 Processor -- start 2024-04-22T01:03:23.941 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.159+0000 7f8fb7088700 1 -- start start 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb7088700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb019bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb7088700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb010c200 con 0x7f8fb0107ca0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb4e24700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb019bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb4e24700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb019bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59490/0 (socket says 172.21.15.38:59490) 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb4e24700 1 -- 172.21.15.38:0/721236159 learned_addr learned my addr 172.21.15.38:0/721236159 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb4e24700 1 -- 172.21.15.38:0/721236159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb019c180 con 0x7f8fb0107ca0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb4e24700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb019bc40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40116a0 tx=0x7f8fa4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fa4003a40 con 0x7f8fb0107ca0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4003be0 con 0x7f8fb0107ca0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb019c410 con 0x7f8fb0107ca0 2024-04-22T01:03:23.942 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.160+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fa401a720 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.161+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb019c870 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.161+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8fa401a9d0 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.162+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb0063890 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.162+0000 7f8fadffb700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fa0039150 0x7f8fa003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.162+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8fa404d8c0 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.162+0000 7f8faffff700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fa0039150 0x7f8fa003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.162+0000 7f8faffff700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fa0039150 0x7f8fa003b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c006fd0 tx=0x7f8f9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.167+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8fa4011e20 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.373+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f8fb019f2e0 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.373+0000 7f8fadffb700 1 -- 172.21.15.38:0/721236159 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f8fa402c350 con 0x7f8fb0107ca0 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fa0039150 msgr2=0x7f8fa003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fa0039150 0x7f8fa003b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c006fd0 tx=0x7f8f9c006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:23.943 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 msgr2=0x7f8fb019bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb019bc40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40116a0 tx=0x7f8fa4011ba0 comp rx=0 tx=0).stop 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 shutdown_connections 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fa0039150 0x7f8fa003b610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 --2- 172.21.15.38:0/721236159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fb0107ca0 0x7f8fb019bc40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 >> 172.21.15.38:0/721236159 conn(0x7f8fb01032a0 msgr2=0x7f8fb0103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 shutdown_connections 2024-04-22T01:03:23.944 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:23.376+0000 7f8fb7088700 1 -- 172.21.15.38:0/721236159 wait complete. 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.359+0000 7f8cf8f0a700 1 Processor -- start 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.359+0000 7f8cf8f0a700 1 -- start start 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.360+0000 7f8cf8f0a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf4108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.360+0000 7f8cf8f0a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41086d0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.360+0000 7f8cf259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf4108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.360+0000 7f8cf259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf4108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59498/0 (socket says 172.21.15.38:59498) 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.360+0000 7f8cf259c700 1 -- 172.21.15.38:0/2222225173 learned_addr learned my addr 172.21.15.38:0/2222225173 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.360+0000 7f8cf259c700 1 -- 172.21.15.38:0/2222225173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf4108e40 con 0x7f8cf4107ce0 2024-04-22T01:03:25.266 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf259c700 1 --2- 172.21.15.38:0/2222225173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf4108100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40098a0 tx=0x7f8ce4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f917a54f1e05cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf159a700 1 -- 172.21.15.38:0/2222225173 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ce4004160 con 0x7f8cf4107ce0 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf159a700 1 -- 172.21.15.38:0/2222225173 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce4004300 con 0x7f8cf4107ce0 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf159a700 1 -- 172.21.15.38:0/2222225173 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ce4011b20 con 0x7f8cf4107ce0 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2222225173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 msgr2=0x7f8cf4108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf8f0a700 1 --2- 172.21.15.38:0/2222225173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf4108100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40098a0 tx=0x7f8ce4009bb0 comp rx=0 tx=0).stop 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2222225173 shutdown_connections 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf8f0a700 1 --2- 172.21.15.38:0/2222225173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf4108100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.361+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2222225173 >> 172.21.15.38:0/2222225173 conn(0x7f8cf4103300 msgr2=0x7f8cf4105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.362+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2222225173 shutdown_connections 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.362+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2222225173 wait complete. 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.362+0000 7f8cf8f0a700 1 Processor -- start 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.362+0000 7f8cf8f0a700 1 -- start start 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf8f0a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:25.267 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf8f0a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf410c1f0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf419bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59510/0 (socket says 172.21.15.38:59510) 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf259c700 1 -- 172.21.15.38:0/2841910742 learned_addr learned my addr 172.21.15.38:0/2841910742 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf259c700 1 -- 172.21.15.38:0/2841910742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf419c0e0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf259c700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf419bba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40117e0 tx=0x7f8ce4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ce4003bf0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce401a910 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf419c370 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ce4024970 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.363+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf419c7d0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.364+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8ce4022020 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.364+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf4063890 con 0x7f8cf4107ce0 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.364+0000 7f8ce37fe700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8cdc0391a0 0x7f8cdc03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:25.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.364+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8ce404d6b0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.365+0000 7f8cf1d9b700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8cdc0391a0 0x7f8cdc03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.365+0000 7f8cf1d9b700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8cdc0391a0 0x7f8cdc03b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8006fd0 tx=0x7f8ce8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.369+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8ce400b3e0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.694+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f8cf419f1a0 con 0x7f8cf4107ce0 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.698+0000 7f8ce37fe700 1 -- 172.21.15.38:0/2841910742 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f8ce4018e00 con 0x7f8cf4107ce0 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8cdc0391a0 msgr2=0x7f8cdc03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8cdc0391a0 0x7f8cdc03b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8006fd0 tx=0x7f8ce8006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 msgr2=0x7f8cf419bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf419bba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40117e0 tx=0x7f8ce4011ba0 comp rx=0 tx=0).stop 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 shutdown_connections 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8cdc0391a0 0x7f8cdc03b660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 --2- 172.21.15.38:0/2841910742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cf4107ce0 0x7f8cf419bba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 >> 172.21.15.38:0/2841910742 conn(0x7f8cf4103300 msgr2=0x7f8cf4103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:25.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 shutdown_connections 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-22T01:03:24.700+0000 7f8cf8f0a700 1 -- 172.21.15.38:0/2841910742 wait complete. 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2024-04-22T01:03:25.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.271 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2024-04-22T01:03:25.271 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.271 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T01:03:25.271 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:25.271 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2024-04-22T01:03:25.302 INFO:tasks.cephadm:Fetching config... 2024-04-22T01:03:25.302 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:03:25.302 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T01:03:25.319 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T01:03:25.319 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:03:25.319 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T01:03:25.375 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T01:03:25.375 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:03:25.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T01:03:25.440 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T01:03:25.440 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:03:25.441 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T01:03:25.496 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T01:03:25.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T01:03:25.574 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:03:25.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T01:03:25.626 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:03:25.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T01:03:25.677 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3efjZCcVf10CIP0/vNGP+A5vsrQ8hMsr79rnBO2zv9i/trPSPc/MDeR8uSjpeLR7WMfIcH8vTEOpc2FYdw5pmznHNzuGAziat6OYbxK91FnDgNw6SSlCSaXDqcGi/3OqMWIVv7wmC9blx3qhhHmMaCMluPErw7wFWCpsO9MYmGk62K3E8QsaD/Ao3VhEyeZzLaUZ8PYKmytdPcZKHcGKcPqBRYUOdoGcwtTeK7GlRk2AOYp5Q+6n4r+6TYVNvijv+wMJqH3pp58IgjGJZ6PXE6ObogIlOI0bfW9RFla/qiv2bt1b4lWRme8aOaG/3jetTKxBEknFBBujkBcC/xvOr9nRPx5fZJvOZ2Y+N6h3ZP+DKiWxttc9caLIlxZcifXqvBlGREW5T+VmDcjAVDJ+SubXMI/C7Csi9o+H7k8+4VnUzJ9519Ofwhrvt8nVncqqqdTq1YGrcp5DVB5ujBagUBXq0fuKZAbsMz/gId6fFW2uuHQnY5+YPQYFvTUmsEe8= ceph-caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:03:25.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T01:03:25.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:25 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2841910742' entity='client.admin' 2024-04-22T01:03:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:28.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 -- 172.21.15.38:0/2582834072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 msgr2=0x7f4cbc100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 --2- 172.21.15.38:0/2582834072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc100fb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009910 tx=0x7f4cac009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 -- 172.21.15.38:0/2582834072 shutdown_connections 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 --2- 172.21.15.38:0/2582834072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc100fb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 -- 172.21.15.38:0/2582834072 >> 172.21.15.38:0/2582834072 conn(0x7f4cbc0fc130 msgr2=0x7f4cbc0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 -- 172.21.15.38:0/2582834072 shutdown_connections 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.698+0000 7f4cc25ef700 1 -- 172.21.15.38:0/2582834072 wait complete. 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.699+0000 7f4cc25ef700 1 Processor -- start 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.699+0000 7f4cc25ef700 1 -- start start 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.699+0000 7f4cc25ef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.699+0000 7f4cc25ef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc106c20 con 0x7f4cbc100b90 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cbbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:28.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cbbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41498/0 (socket says 172.21.15.38:41498) 2024-04-22T01:03:28.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cbbfff700 1 -- 172.21.15.38:0/1896343344 learned_addr learned my addr 172.21.15.38:0/1896343344 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cbbfff700 1 -- 172.21.15.38:0/1896343344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc1978d0 con 0x7f4cbc100b90 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cbbfff700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc197390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006010 tx=0x7f4cac005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cac027070 con 0x7f4cbc100b90 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac00be40 con 0x7f4cbc100b90 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc197b60 con 0x7f4cbc100b90 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cac00f460 con 0x7f4cbc100b90 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.700+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc19a850 con 0x7f4cbc100b90 2024-04-22T01:03:28.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.701+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4cac02a070 con 0x7f4cbc100b90 2024-04-22T01:03:28.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.701+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cbc1073a0 con 0x7f4cbc100b90 2024-04-22T01:03:28.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.701+0000 7f4cb97fa700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ca4039150 0x7f4ca403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:28.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.701+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4cac01e030 con 0x7f4cbc100b90 2024-04-22T01:03:28.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.701+0000 7f4cbb7fe700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ca4039150 0x7f4ca403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:28.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.702+0000 7f4cbb7fe700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ca4039150 0x7f4ca403b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006fd0 tx=0x7f4cb0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:28.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.706+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4cac013070 con 0x7f4cbc100b90 2024-04-22T01:03:28.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.916+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f4cbc02dc40 con 0x7f4cbc100b90 2024-04-22T01:03:28.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.919+0000 7f4cb97fa700 1 -- 172.21.15.38:0/1896343344 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4cac0310c0 con 0x7f4cbc100b90 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.921+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ca4039150 msgr2=0x7f4ca403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.921+0000 7f4cc25ef700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ca4039150 0x7f4ca403b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0006fd0 tx=0x7f4cb0006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.921+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 msgr2=0x7f4cbc197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.921+0000 7f4cc25ef700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc197390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006010 tx=0x7f4cac005050 comp rx=0 tx=0).stop 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.922+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 shutdown_connections 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.922+0000 7f4cc25ef700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ca4039150 0x7f4ca403b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.922+0000 7f4cc25ef700 1 --2- 172.21.15.38:0/1896343344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cbc100b90 0x7f4cbc197390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.922+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 >> 172.21.15.38:0/1896343344 conn(0x7f4cbc0fc130 msgr2=0x7f4cbc069a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.922+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 shutdown_connections 2024-04-22T01:03:28.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:28.922+0000 7f4cc25ef700 1 -- 172.21.15.38:0/1896343344 wait complete. 2024-04-22T01:03:29.118 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:29.118 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:29.118 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:03:29.118 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:29.118 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:29.118 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:30.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:29 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1896343344' entity='client.admin' 2024-04-22T01:03:30.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:30.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:30.163 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T01:03:30.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T01:03:30.412 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 -- 172.21.15.38:0/2799293743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 msgr2=0x7f095c1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 --2- 172.21.15.38:0/2799293743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1020b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f094c009910 tx=0x7f094c009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 -- 172.21.15.38:0/2799293743 shutdown_connections 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 --2- 172.21.15.38:0/2799293743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1020b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 -- 172.21.15.38:0/2799293743 >> 172.21.15.38:0/2799293743 conn(0x7f095c0fd230 msgr2=0x7f095c0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 -- 172.21.15.38:0/2799293743 shutdown_connections 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.977+0000 7f0962a70700 1 -- 172.21.15.38:0/2799293743 wait complete. 2024-04-22T01:03:31.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f0962a70700 1 Processor -- start 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f0962a70700 1 -- start start 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f0962a70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f0962a70700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f095c107d20 con 0x7f095c101c90 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f096080c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f096080c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41528/0 (socket says 172.21.15.38:41528) 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f096080c700 1 -- 172.21.15.38:0/3239200747 learned_addr learned my addr 172.21.15.38:0/3239200747 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f096080c700 1 -- 172.21.15.38:0/3239200747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c1978e0 con 0x7f095c101c90 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.978+0000 7f096080c700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1973a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f094c006010 tx=0x7f094c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.979+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f094c027070 con 0x7f095c101c90 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.979+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f094c00be40 con 0x7f095c101c90 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.979+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c197b70 con 0x7f095c101c90 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.979+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f094c00f460 con 0x7f095c101c90 2024-04-22T01:03:31.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.979+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c19a860 con 0x7f095c101c90 2024-04-22T01:03:31.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.980+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f094c02a070 con 0x7f095c101c90 2024-04-22T01:03:31.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.980+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c1084a0 con 0x7f095c101c90 2024-04-22T01:03:31.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.980+0000 7f0959ffb700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0944039150 0x7f094403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:31.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.980+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f094c01e030 con 0x7f095c101c90 2024-04-22T01:03:31.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.980+0000 7f095bfff700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0944039150 0x7f094403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:31.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.981+0000 7f095bfff700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0944039150 0x7f094403b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0950006fd0 tx=0x7f0950006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:31.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:31.985+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f094c013070 con 0x7f095c101c90 2024-04-22T01:03:32.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.196+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f095c02dc40 con 0x7f0944039150 2024-04-22T01:03:32.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.199+0000 7f0959ffb700 1 -- 172.21.15.38:0/3239200747 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f095c02dc40 con 0x7f0944039150 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.201+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0944039150 msgr2=0x7f094403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.201+0000 7f0962a70700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0944039150 0x7f094403b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0950006fd0 tx=0x7f0950006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.201+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 msgr2=0x7f095c1973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.201+0000 7f0962a70700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1973a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f094c006010 tx=0x7f094c005050 comp rx=0 tx=0).stop 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.202+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 shutdown_connections 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.202+0000 7f0962a70700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0944039150 0x7f094403b610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.202+0000 7f0962a70700 1 --2- 172.21.15.38:0/3239200747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f095c101c90 0x7f095c1973a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.202+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 >> 172.21.15.38:0/3239200747 conn(0x7f095c0fd230 msgr2=0x7f095c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:32.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.202+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 shutdown_connections 2024-04-22T01:03:32.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:32.202+0000 7f0962a70700 1 -- 172.21.15.38:0/3239200747 wait complete. 2024-04-22T01:03:33.004 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi111 2024-04-22T01:03:33.005 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:03:33.005 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.conf 2024-04-22T01:03:33.024 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:03:33.024 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:03:33.081 INFO:tasks.cephadm:Adding host smithi111 to orchestrator... 2024-04-22T01:03:33.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch host add smithi111 2024-04-22T01:03:33.323 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: Updating smithi038:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:03:33.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:33.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:33.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:34.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:34 smithi038 ceph-mon[28043]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:03:34.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:34 smithi038 ceph-mon[28043]: Updating smithi038:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 -- 172.21.15.38:0/4143603940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 msgr2=0x7f62c0106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 --2- 172.21.15.38:0/4143603940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c0106fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009910 tx=0x7f62b0009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 -- 172.21.15.38:0/4143603940 shutdown_connections 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 --2- 172.21.15.38:0/4143603940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c0106fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 -- 172.21.15.38:0/4143603940 >> 172.21.15.38:0/4143603940 conn(0x7f62c00fc630 msgr2=0x7f62c00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 -- 172.21.15.38:0/4143603940 shutdown_connections 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 -- 172.21.15.38:0/4143603940 wait complete. 2024-04-22T01:03:34.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.842+0000 7f62c5e08700 1 Processor -- start 2024-04-22T01:03:34.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62c5e08700 1 -- start start 2024-04-22T01:03:34.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62c5e08700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:34.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62c5e08700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62c010b1b0 con 0x7f62c0106bf0 2024-04-22T01:03:34.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:34.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41548/0 (socket says 172.21.15.38:41548) 2024-04-22T01:03:34.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62bf7fe700 1 -- 172.21.15.38:0/1397524247 learned_addr learned my addr 172.21.15.38:0/1397524247 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:34.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62bf7fe700 1 -- 172.21.15.38:0/1397524247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c019fbd0 con 0x7f62c0106bf0 2024-04-22T01:03:34.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.843+0000 7f62bf7fe700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c019f690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f62b0005850 tx=0x7f62b0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:34.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.844+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62b0026070 con 0x7f62c0106bf0 2024-04-22T01:03:34.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.844+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62b000bc30 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.844+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62b0003a20 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.844+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c019fe60 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.844+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c01997b0 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.844+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f62b0029070 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.845+0000 7f62bcff9700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f62a8039150 0x7f62a803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.845+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f62b001e070 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.845+0000 7f62beffd700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f62a8039150 0x7f62a803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.845+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c01021f0 con 0x7f62c0106bf0 2024-04-22T01:03:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.845+0000 7f62beffd700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f62a8039150 0x7f62a803b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f62b4006fd0 tx=0x7f62b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:34.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:34.850+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f62b0030070 con 0x7f62c0106bf0 2024-04-22T01:03:35.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:35.060+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}) -- 0x7f62c0063890 con 0x7f62a8039150 2024-04-22T01:03:36.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:35 smithi038 ceph-mon[28043]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:36.965 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:36 smithi038 ceph-mon[28043]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:36.965 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:36 smithi038 ceph-mon[28043]: Deploying cephadm binary to smithi111 2024-04-22T01:03:39.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:38 smithi038 ceph-mon[28043]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:39.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.659+0000 7f62bcff9700 1 -- 172.21.15.38:0/1397524247 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f62c0063890 con 0x7f62a8039150 2024-04-22T01:03:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi111' with addr '172.21.15.111' 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f62a8039150 msgr2=0x7f62a803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f62a8039150 0x7f62a803b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f62b4006fd0 tx=0x7f62b4006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 msgr2=0x7f62c019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c019f690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f62b0005850 tx=0x7f62b0005030 comp rx=0 tx=0).stop 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 shutdown_connections 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f62a8039150 0x7f62a803b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 --2- 172.21.15.38:0/1397524247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62c0106bf0 0x7f62c019f690 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:39.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 >> 172.21.15.38:0/1397524247 conn(0x7f62c00fc630 msgr2=0x7f62c00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:39.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 shutdown_connections 2024-04-22T01:03:39.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:39.662+0000 7f62c5e08700 1 -- 172.21.15.38:0/1397524247 wait complete. 2024-04-22T01:03:40.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T01:03:40.560 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:41.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:41.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:40 smithi038 ceph-mon[28043]: Added host smithi111 2024-04-22T01:03:41.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:41.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:40 smithi038 ceph-mon[28043]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:41.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:41 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1737460917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 msgr2=0x7f84d4106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 --2- 172.21.15.38:0/1737460917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4106740 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f84c400aa20 tx=0x7f84c400ad30 comp rx=0 tx=0).stop 2024-04-22T01:03:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1737460917 shutdown_connections 2024-04-22T01:03:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 --2- 172.21.15.38:0/1737460917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4106740 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1737460917 >> 172.21.15.38:0/1737460917 conn(0x7f84d40780c0 msgr2=0x7f84d40784d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1737460917 shutdown_connections 2024-04-22T01:03:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.166+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1737460917 wait complete. 2024-04-22T01:03:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d9ff0700 1 Processor -- start 2024-04-22T01:03:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d9ff0700 1 -- start start 2024-04-22T01:03:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d9ff0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d9ff0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84d410a070 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48584/0 (socket says 172.21.15.38:48584) 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.167+0000 7f84d37fe700 1 -- 172.21.15.38:0/1507159061 learned_addr learned my addr 172.21.15.38:0/1507159061 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d37fe700 1 -- 172.21.15.38:0/1507159061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d4197c50 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d37fe700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4197710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c6d0 tx=0x7f84c4004d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40053a0 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c4005540 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d4197ee0 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c400e6c0 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.168+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d419bfe0 con 0x7f84d4106360 2024-04-22T01:03:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.169+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f84c400e920 con 0x7f84d4106360 2024-04-22T01:03:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.169+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d4101a10 con 0x7f84d4106360 2024-04-22T01:03:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.169+0000 7f84d0ff9700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84bc0390f0 0x7f84bc03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.169+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f84c401e070 con 0x7f84d4106360 2024-04-22T01:03:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.169+0000 7f84d2ffd700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84bc0390f0 0x7f84bc03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.170+0000 7f84d2ffd700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84bc0390f0 0x7f84bc03b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f84c8006fd0 tx=0x7f84c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:42.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.174+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f84c4014e20 con 0x7f84d4106360 2024-04-22T01:03:42.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.389+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f84d4063890 con 0x7f84bc0390f0 2024-04-22T01:03:42.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.391+0000 7f84d0ff9700 1 -- 172.21.15.38:0/1507159061 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f84d4063890 con 0x7f84bc0390f0 2024-04-22T01:03:42.391 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:42.391 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}] 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84bc0390f0 msgr2=0x7f84bc03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84bc0390f0 0x7f84bc03b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f84c8006fd0 tx=0x7f84c8006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 msgr2=0x7f84d4197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4197710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c6d0 tx=0x7f84c4004d80 comp rx=0 tx=0).stop 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 shutdown_connections 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84bc0390f0 0x7f84bc03b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 --2- 172.21.15.38:0/1507159061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d4106360 0x7f84d4197710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 >> 172.21.15.38:0/1507159061 conn(0x7f84d40780c0 msgr2=0x7f84d4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 shutdown_connections 2024-04-22T01:03:42.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:42.393+0000 7f84d9ff0700 1 -- 172.21.15.38:0/1507159061 wait complete. 2024-04-22T01:03:43.016 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-04-22T01:03:43.016 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:03:43.016 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-04-22T01:03:43.035 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:03:43.035 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:03:43.094 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-04-22T01:03:43.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch host add smithi203 2024-04-22T01:03:43.346 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:43.563 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:43 smithi038 ceph-mon[28043]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:43.563 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:43.563 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:43 smithi038 ceph-mon[28043]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 -- 172.21.15.38:0/1003465921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 msgr2=0x7f118c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 --2- 172.21.15.38:0/1003465921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c106fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f117c009910 tx=0x7f117c009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 -- 172.21.15.38:0/1003465921 shutdown_connections 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 --2- 172.21.15.38:0/1003465921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c106fe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 -- 172.21.15.38:0/1003465921 >> 172.21.15.38:0/1003465921 conn(0x7f118c076690 msgr2=0x7f118c076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 -- 172.21.15.38:0/1003465921 shutdown_connections 2024-04-22T01:03:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.866+0000 7f1192817700 1 -- 172.21.15.38:0/1003465921 wait complete. 2024-04-22T01:03:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f1192817700 1 Processor -- start 2024-04-22T01:03:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f1192817700 1 -- start start 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f1192817700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c1027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f1192817700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f118c10b1c0 con 0x7f118c106c00 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f118bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c1027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f118bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c1027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48614/0 (socket says 172.21.15.38:48614) 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.867+0000 7f118bfff700 1 -- 172.21.15.38:0/798487291 learned_addr learned my addr 172.21.15.38:0/798487291 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f118bfff700 1 -- 172.21.15.38:0/798487291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c1046b0 con 0x7f118c106c00 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f118bfff700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c1027c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f117c005850 tx=0x7f117c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f117c026070 con 0x7f118c106c00 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f117c00bc30 con 0x7f118c106c00 2024-04-22T01:03:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118c102d00 con 0x7f118c106c00 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f117c003a20 con 0x7f118c106c00 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.868+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118c1031c0 con 0x7f118c106c00 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.869+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f117c029070 con 0x7f118c106c00 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.869+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118c102200 con 0x7f118c106c00 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.869+0000 7f11897fa700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1174039150 0x7f117403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.869+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f117c01e070 con 0x7f118c106c00 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.870+0000 7f118b7fe700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1174039150 0x7f117403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.870+0000 7f118b7fe700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1174039150 0x7f117403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1180006fd0 tx=0x7f1180006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:44.874+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f117c013070 con 0x7f118c106c00 2024-04-22T01:03:45.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:45.088+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) -- 0x7f118c103d60 con 0x7f1174039150 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:03:45.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:03:46.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:46 smithi038 ceph-mon[28043]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:46 smithi038 ceph-mon[28043]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T01:03:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:46 smithi038 ceph-mon[28043]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:03:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:47.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:47 smithi038 ceph-mon[28043]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:03:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:47 smithi038 ceph-mon[28043]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:47 smithi038 ceph-mon[28043]: Deploying cephadm binary to smithi203 2024-04-22T01:03:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:47 smithi038 ceph-mon[28043]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T01:03:49.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.587+0000 7f11897fa700 1 -- 172.21.15.38:0/798487291 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f118c103d60 con 0x7f1174039150 2024-04-22T01:03:49.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:49 smithi038 ceph-mon[28043]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:49.587 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1174039150 msgr2=0x7f117403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1174039150 0x7f117403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1180006fd0 tx=0x7f1180006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 msgr2=0x7f118c1027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c1027c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f117c005850 tx=0x7f117c005030 comp rx=0 tx=0).stop 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 shutdown_connections 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1174039150 0x7f117403b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:49.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 --2- 172.21.15.38:0/798487291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c106c00 0x7f118c1027c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:49.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 >> 172.21.15.38:0/798487291 conn(0x7f118c076690 msgr2=0x7f118c0fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:49.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 shutdown_connections 2024-04-22T01:03:49.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:49.590+0000 7f1192817700 1 -- 172.21.15.38:0/798487291 wait complete. 2024-04-22T01:03:50.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T01:03:50.426 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:50.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:50.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:50 smithi038 ceph-mon[28043]: Added host smithi203 2024-04-22T01:03:50.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:50.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:51.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:51 smithi038 ceph-mon[28043]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:51.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:51 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 -- 172.21.15.38:0/1470597338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 msgr2=0x7f38001022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 --2- 172.21.15.38:0/1470597338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f38001022b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009910 tx=0x7f37f0009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 -- 172.21.15.38:0/1470597338 shutdown_connections 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 --2- 172.21.15.38:0/1470597338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f38001022b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 -- 172.21.15.38:0/1470597338 >> 172.21.15.38:0/1470597338 conn(0x7f38000fd760 msgr2=0x7f38000ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 -- 172.21.15.38:0/1470597338 shutdown_connections 2024-04-22T01:03:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 -- 172.21.15.38:0/1470597338 wait complete. 2024-04-22T01:03:52.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.000+0000 7f38050f8700 1 Processor -- start 2024-04-22T01:03:52.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f38050f8700 1 -- start start 2024-04-22T01:03:52.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f38050f8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f380019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:52.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f38050f8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f380010c2a0 con 0x7f3800101ed0 2024-04-22T01:03:52.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f37fed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f380019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:52.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f37fed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f380019d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54636/0 (socket says 172.21.15.38:54636) 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f37fed9d700 1 -- 172.21.15.38:0/1905757611 learned_addr learned my addr 172.21.15.38:0/1905757611 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f37fed9d700 1 -- 172.21.15.38:0/1905757611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380019d950 con 0x7f3800101ed0 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f37fed9d700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f380019d410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37f0006010 tx=0x7f37f0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.001+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f0026070 con 0x7f3800101ed0 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.002+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f000bbd0 con 0x7f3800101ed0 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.002+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380019dc40 con 0x7f3800101ed0 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.002+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f0003910 con 0x7f3800101ed0 2024-04-22T01:03:52.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.002+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38001975d0 con 0x7f3800101ed0 2024-04-22T01:03:52.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.003+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f37f0029070 con 0x7f3800101ed0 2024-04-22T01:03:52.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.003+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38001033b0 con 0x7f3800101ed0 2024-04-22T01:03:52.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.003+0000 7f37fcd99700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37ec039150 0x7f37ec03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:52.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.003+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f37f001e070 con 0x7f3800101ed0 2024-04-22T01:03:52.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.003+0000 7f37f7fff700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37ec039150 0x7f37ec03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:52.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.003+0000 7f37f7fff700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37ec039150 0x7f37ec03b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f37e8006fd0 tx=0x7f37e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:52.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.008+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f37f0050030 con 0x7f3800101ed0 2024-04-22T01:03:52.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.219+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3800198b00 con 0x7f37ec039150 2024-04-22T01:03:52.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.221+0000 7f37fcd99700 1 -- 172.21.15.38:0/1905757611 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f3800198b00 con 0x7f37ec039150 2024-04-22T01:03:52.220 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:03:52.220 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-04-22T01:03:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37ec039150 msgr2=0x7f37ec03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37ec039150 0x7f37ec03b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f37e8006fd0 tx=0x7f37e8006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 msgr2=0x7f380019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f380019d410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37f0006010 tx=0x7f37f0005030 comp rx=0 tx=0).stop 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 shutdown_connections 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37ec039150 0x7f37ec03b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 --2- 172.21.15.38:0/1905757611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3800101ed0 0x7f380019d410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 >> 172.21.15.38:0/1905757611 conn(0x7f38000fd760 msgr2=0x7f3800104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 shutdown_connections 2024-04-22T01:03:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:52.223+0000 7f38050f8700 1 -- 172.21.15.38:0/1905757611 wait complete. 2024-04-22T01:03:52.853 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T01:03:52.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T01:03:53.098 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:03:53.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:53 smithi038 ceph-mon[28043]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:53.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:53.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:53 smithi038 ceph-mon[28043]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 -- 172.21.15.38:0/1649982622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 msgr2=0x7efff81080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 --2- 172.21.15.38:0/1649982622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff81080b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7effe8009910 tx=0x7effe8009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 -- 172.21.15.38:0/1649982622 shutdown_connections 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 --2- 172.21.15.38:0/1649982622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff81080b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 -- 172.21.15.38:0/1649982622 >> 172.21.15.38:0/1649982622 conn(0x7efff8076690 msgr2=0x7efff8076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 -- 172.21.15.38:0/1649982622 shutdown_connections 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.703+0000 7efffd523700 1 -- 172.21.15.38:0/1649982622 wait complete. 2024-04-22T01:03:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efffd523700 1 Processor -- start 2024-04-22T01:03:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efffd523700 1 -- start start 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efffd523700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efffd523700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efff810c260 con 0x7efff8107cd0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efff6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efff6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54664/0 (socket says 172.21.15.38:54664) 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.704+0000 7efff6ffd700 1 -- 172.21.15.38:0/224248922 learned_addr learned my addr 172.21.15.38:0/224248922 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7efff6ffd700 1 -- 172.21.15.38:0/224248922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff819d910 con 0x7efff8107cd0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7efff6ffd700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff819d3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7effe8006010 tx=0x7effe8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effe8026070 con 0x7efff8107cd0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff819dc00 con 0x7efff8107cd0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe800bbd0 con 0x7efff8107cd0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effe8003910 con 0x7efff8107cd0 2024-04-22T01:03:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.705+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff8197590 con 0x7efff8107cd0 2024-04-22T01:03:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.706+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7effe8029070 con 0x7efff8107cd0 2024-04-22T01:03:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.706+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff8103380 con 0x7efff8107cd0 2024-04-22T01:03:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.706+0000 7effdffff700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00390f0 0x7effe003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.706+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7effe801e070 con 0x7efff8107cd0 2024-04-22T01:03:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.706+0000 7efff67fc700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00390f0 0x7effe003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.707+0000 7efff67fc700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00390f0 0x7effe003b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7effec006fd0 tx=0x7effec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:54.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.711+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7effe8013070 con 0x7efff8107cd0 2024-04-22T01:03:54.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:54.921+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7efff8063890 con 0x7efff8107cd0 2024-04-22T01:03:55.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.163+0000 7effdffff700 1 -- 172.21.15.38:0/224248922 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7effe8030090 con 0x7efff8107cd0 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00390f0 msgr2=0x7effe003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00390f0 0x7effe003b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7effec006fd0 tx=0x7effec006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 msgr2=0x7efff819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff819d3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7effe8006010 tx=0x7effe8005030 comp rx=0 tx=0).stop 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 shutdown_connections 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00390f0 0x7effe003b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 --2- 172.21.15.38:0/224248922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8107cd0 0x7efff819d3d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:55.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 >> 172.21.15.38:0/224248922 conn(0x7efff8076690 msgr2=0x7efff80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:55.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 shutdown_connections 2024-04-22T01:03:55.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:03:55.165+0000 7efffd523700 1 -- 172.21.15.38:0/224248922 wait complete. 2024-04-22T01:03:55.165 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-04-22T01:03:55.177 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:55 smithi038 ceph-mon[28043]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:55.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:55 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/224248922' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T01:03:55.829 INFO:tasks.cephadm:Adding mon.a on smithi038 2024-04-22T01:03:55.829 INFO:tasks.cephadm:Adding mon.b on smithi111 2024-04-22T01:03:55.829 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-04-22T01:03:55.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c' 2024-04-22T01:03:56.071 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/224248922' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:03:56.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:03:56.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:03:56.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T01:03:56.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:56 smithi038 ceph-mon[28043]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:03:57.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.620+0000 7f98538ec700 1 -- 172.21.15.203:0/977204198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 msgr2=0x7f984c1081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.620+0000 7f98538ec700 1 --2- 172.21.15.203:0/977204198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c1081a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9844009910 tx=0x7f9844009c20 comp rx=0 tx=0).stop 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 -- 172.21.15.203:0/977204198 shutdown_connections 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 --2- 172.21.15.203:0/977204198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c1081a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 -- 172.21.15.203:0/977204198 >> 172.21.15.203:0/977204198 conn(0x7f984c0772b0 msgr2=0x7f984c0776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 -- 172.21.15.203:0/977204198 shutdown_connections 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 -- 172.21.15.203:0/977204198 wait complete. 2024-04-22T01:03:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 Processor -- start 2024-04-22T01:03:57.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 -- start start 2024-04-22T01:03:57.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:57.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.621+0000 7f98538ec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10c3c0 con 0x7f984c107dc0 2024-04-22T01:03:57.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.622+0000 7f9851688700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:57.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.622+0000 7f9851688700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.203:44594/0 (socket says 172.21.15.203:44594) 2024-04-22T01:03:57.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.622+0000 7f9851688700 1 -- 172.21.15.203:0/647798192 learned_addr learned my addr 172.21.15.203:0/647798192 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:03:57.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.622+0000 7f9851688700 1 -- 172.21.15.203:0/647798192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984c19daf0 con 0x7f984c107dc0 2024-04-22T01:03:57.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.622+0000 7f9851688700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c19d5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9844006010 tx=0x7f9844005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:57.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.622+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9844026070 con 0x7f984c107dc0 2024-04-22T01:03:57.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.623+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984400bc30 con 0x7f984c107dc0 2024-04-22T01:03:57.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.623+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98440039d0 con 0x7f984c107dc0 2024-04-22T01:03:57.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.623+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984c19dd80 con 0x7f984c107dc0 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.623+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f984c1976d0 con 0x7f984c107dc0 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.623+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f984c1034e0 con 0x7f984c107dc0 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.623+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9844029070 con 0x7f984c107dc0 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.624+0000 7f98427fc700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9838039150 0x7f983803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.624+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f984401e070 con 0x7f984c107dc0 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.624+0000 7f9850e87700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9838039150 0x7f983803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:03:57.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.624+0000 7f9850e87700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9838039150 0x7f983803b610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f983c006fd0 tx=0x7f983c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:03:57.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.629+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9844013070 con 0x7f984c107dc0 2024-04-22T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:57 smithi038 ceph-mon[28043]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:57 smithi038 ceph-mon[28043]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:57 smithi038 ceph-mon[28043]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:57.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.836+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) -- 0x7f984c198bb0 con 0x7f9838039150 2024-04-22T01:03:57.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.846+0000 7f98427fc700 1 -- 172.21.15.203:0/647798192 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f984c198bb0 con 0x7f9838039150 2024-04-22T01:03:57.845 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-04-22T01:03:57.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9838039150 msgr2=0x7f983803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9838039150 0x7f983803b610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f983c006fd0 tx=0x7f983c006e40 comp rx=0 tx=0).stop 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 msgr2=0x7f984c19d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c19d5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9844006010 tx=0x7f9844005030 comp rx=0 tx=0).stop 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 shutdown_connections 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9838039150 0x7f983803b610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 --2- 172.21.15.203:0/647798192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f984c107dc0 0x7f984c19d5b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 >> 172.21.15.203:0/647798192 conn(0x7f984c0772b0 msgr2=0x7f984c103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 shutdown_connections 2024-04-22T01:03:57.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:03:57.848+0000 7f98538ec700 1 -- 172.21.15.203:0/647798192 wait complete. 2024-04-22T01:03:58.453 DEBUG:teuthology.orchestra.run.smithi111:mon.b> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T01:03:58.456 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T01:03:58.459 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T01:03:58.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c;count:3 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:03:59.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:03:59.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:03:58 smithi038 ceph-mon[28043]: Deploying daemon mon.c on smithi203 2024-04-22T01:04:00.412 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:04:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:00 smithi038 ceph-mon[28043]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 -- 172.21.15.203:0/3834157190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 msgr2=0x7f72c0101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 --2- 172.21.15.203:0/3834157190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0101260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f72b800b0a0 tx=0x7f72b800b3b0 comp rx=0 tx=0).stop 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 -- 172.21.15.203:0/3834157190 shutdown_connections 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 --2- 172.21.15.203:0/3834157190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0101260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 -- 172.21.15.203:0/3834157190 >> 172.21.15.203:0/3834157190 conn(0x7f72c00fc710 msgr2=0x7f72c00feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 -- 172.21.15.203:0/3834157190 shutdown_connections 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 -- 172.21.15.203:0/3834157190 wait complete. 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.297+0000 7f72c5fb1700 1 Processor -- start 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72c5fb1700 1 -- start start 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72c5fb1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:02.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72c5fb1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010b250 con 0x7f72c0100e80 2024-04-22T01:04:02.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:02.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.203:37490/0 (socket says 172.21.15.203:37490) 2024-04-22T01:04:02.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72bf7fe700 1 -- 172.21.15.203:0/128561263 learned_addr learned my addr 172.21.15.203:0/128561263 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:04:02.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72bf7fe700 1 -- 172.21.15.203:0/128561263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c01047c0 con 0x7f72c0100e80 2024-04-22T01:04:02.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.298+0000 7f72bf7fe700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0102920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f72b8015040 tx=0x7f72b8003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:02.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.299+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72b8017040 con 0x7f72c0100e80 2024-04-22T01:04:02.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.299+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c0102e60 con 0x7f72c0100e80 2024-04-22T01:04:02.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.299+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b8022070 con 0x7f72c0100e80 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.299+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72b801d5d0 con 0x7f72c0100e80 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.299+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c0103320 con 0x7f72c0100e80 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.300+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f72b8004000 con 0x7f72c0100e80 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.300+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c0102360 con 0x7f72c0100e80 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.300+0000 7f72bcff9700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f72ac0390f0 0x7f72ac03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.300+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f72b800f050 con 0x7f72c0100e80 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.300+0000 7f72beffd700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f72ac0390f0 0x7f72ac03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:02.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.301+0000 7f72beffd700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f72ac0390f0 0x7f72ac03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f72a8006fd0 tx=0x7f72a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:02.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.305+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f72b8007e20 con 0x7f72c0100e80 2024-04-22T01:04:02.607 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.607+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f72c0103e50 con 0x7f72c0100e80 2024-04-22T01:04:02.608 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.608+0000 7f72bcff9700 1 -- 172.21.15.203:0/128561263 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f72b8004440 con 0x7f72c0100e80 2024-04-22T01:04:02.608 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:04:02.608 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","modified":"2024-04-22T01:02:16.910765Z","created":"2024-04-22T01:02:16.910765Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f72ac0390f0 msgr2=0x7f72ac03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f72ac0390f0 0x7f72ac03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f72a8006fd0 tx=0x7f72a8006e40 comp rx=0 tx=0).stop 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 msgr2=0x7f72c0102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0102920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f72b8015040 tx=0x7f72b8003d50 comp rx=0 tx=0).stop 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 shutdown_connections 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f72ac0390f0 0x7f72ac03b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 --2- 172.21.15.203:0/128561263 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72c0100e80 0x7f72c0102920 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:02.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 >> 172.21.15.203:0/128561263 conn(0x7f72c00fc710 msgr2=0x7f72c0069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:02.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 shutdown_connections 2024-04-22T01:04:02.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:02.610+0000 7f72c5fb1700 1 -- 172.21.15.203:0/128561263 wait complete. 2024-04-22T01:04:02.611 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-22T01:04:04.496 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T01:04:04.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T01:04:04.742 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:04:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:08.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:08.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: mon.a calling monitor election 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: mon.c calling monitor election 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T01:04:08.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: monmap epoch 2 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: last_changed 2024-04-22T01:04:02.635331+0000 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: min_mon_release 19 (squid) 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: election_strategy: 1 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: fsmap 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: mgrmap e13: x(active, since 51s) 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: overall HEALTH_OK 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:08.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:08.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:08.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:08.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:08 smithi203 ceph-mon[31825]: Deploying daemon mon.b on smithi111 2024-04-22T01:04:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:08 smithi203 ceph-mon[31825]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:09.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 -- 172.21.15.203:0/197354284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0180035c0 msgr2=0x7fa018005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:09.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 --2- 172.21.15.203:0/197354284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0180035c0 0x7fa018005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa028101c70 tx=0x7fa01c00feb0 comp rx=0 tx=0).stop 2024-04-22T01:04:09.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 -- 172.21.15.203:0/197354284 shutdown_connections 2024-04-22T01:04:09.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 --2- 172.21.15.203:0/197354284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0180035c0 0x7fa018005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 -- 172.21.15.203:0/197354284 >> 172.21.15.203:0/197354284 conn(0x7fa0280fd830 msgr2=0x7fa0280ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 -- 172.21.15.203:0/197354284 shutdown_connections 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.064+0000 7fa02efd7700 1 -- 172.21.15.203:0/197354284 wait complete. 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02efd7700 1 Processor -- start 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02efd7700 1 -- start start 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02efd7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02efd7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa02819db70 0x7fa028197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02efd7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa028109dd0 con 0x7fa02819db70 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02efd7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa028109c50 con 0x7fa0180035c0 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51722/0 (socket says 172.21.15.203:51722) 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa027fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa02819db70 0x7fa028197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:09.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 -- 172.21.15.203:0/1896725288 learned_addr learned my addr 172.21.15.203:0/1896725288 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 -- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 msgr2=0x7fa02819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 -- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 msgr2=0x7fa02819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa02cd73700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa027fff700 1 -- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 msgr2=0x7fa02819d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa027fff700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.065+0000 7fa027fff700 1 -- 172.21.15.203:0/1896725288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02819e0f0 con 0x7fa02819db70 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.066+0000 7fa027fff700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa02819db70 0x7fa028197700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa01400b4d0 tx=0x7fa01400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.066+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fa014011630 con 0x7fa02819db70 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.066+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa014011cb0 con 0x7fa02819db70 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.066+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa028197ca0 con 0x7fa02819db70 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.066+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fa014010e60 con 0x7fa02819db70 2024-04-22T01:04:09.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.066+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa028198230 con 0x7fa02819db70 2024-04-22T01:04:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.067+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa014018020 con 0x7fa02819db70 2024-04-22T01:04:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.067+0000 7fa025ffb700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa00c038e40 0x7fa00c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.067+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa01404be50 con 0x7fa02819db70 2024-04-22T01:04:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.067+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa028103480 con 0x7fa02819db70 2024-04-22T01:04:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.067+0000 7fa02cd73700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa00c038e40 0x7fa00c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:09.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.068+0000 7fa02cd73700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa00c038e40 0x7fa00c03b300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00fa60 tx=0x7fa01c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:09.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.077+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa01401f020 con 0x7fa02819db70 2024-04-22T01:04:09.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:08 smithi038 ceph-mon[28043]: Deploying daemon mon.b on smithi111 2024-04-22T01:04:09.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:08 smithi038 ceph-mon[28043]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:09.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:09.352 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.351+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa0281985a0 con 0x7fa02819db70 2024-04-22T01:04:09.352 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.352+0000 7fa025ffb700 1 -- 172.21.15.203:0/1896725288 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fa014018310 con 0x7fa02819db70 2024-04-22T01:04:09.352 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:04:09.352 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","modified":"2024-04-22T01:04:02.635331Z","created":"2024-04-22T01:02:16.910765Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T01:04:09.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa00c038e40 msgr2=0x7fa00c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:09.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa00c038e40 0x7fa00c03b300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00fa60 tx=0x7fa01c005960 comp rx=0 tx=0).stop 2024-04-22T01:04:09.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa02819db70 msgr2=0x7fa028197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:09.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa02819db70 0x7fa028197700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa01400b4d0 tx=0x7fa01400b890 comp rx=0 tx=0).stop 2024-04-22T01:04:09.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 shutdown_connections 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa00c038e40 0x7fa00c03b300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0180035c0 0x7fa02819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 --2- 172.21.15.203:0/1896725288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa02819db70 0x7fa028197700 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 >> 172.21.15.203:0/1896725288 conn(0x7fa0280fd830 msgr2=0x7fa0280feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 shutdown_connections 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:09.354+0000 7fa02efd7700 1 -- 172.21.15.203:0/1896725288 wait complete. 2024-04-22T01:04:09.355 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-04-22T01:04:09.926 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:09 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/1896725288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T01:04:10.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:09 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/1896725288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T01:04:10.954 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T01:04:10.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T01:04:10.981 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:10 smithi203 ceph-mon[31825]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:10 smithi038 ceph-mon[28043]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.199 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:04:11.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 podman[31038]: 2024-04-22 01:04:11.238841266 +0000 UTC m=+0.024092754 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 podman[31038]: 2024-04-22 01:04:11.542824396 +0000 UTC m=+0.328075884 container init 1e22d75ebb5d468af3bfa99119864d4424adbdb1150ddb8f45064fb2045b17e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 podman[31038]: 2024-04-22 01:04:11.545602319 +0000 UTC m=+0.330853808 container start 1e22d75ebb5d468af3bfa99119864d4424adbdb1150ddb8f45064fb2045b17e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev), process ceph-mon, pid 2 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pidfile_write: ignore empty --pid-file 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: load: jerasure load: lrc 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: RocksDB version: 7.9.2 2024-04-22T01:04:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Git sha 0 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Compile date 2024-04-20 15:04:30 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: DB SUMMARY 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: DB Session ID: D3PIQRAS3QGEP39Z5WNG 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: CURRENT file: CURRENT 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-22T01:04:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.error_if_exists: 0 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.create_if_missing: 0 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.paranoid_checks: 1 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.env: 0x5649f09c2c60 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.info_log: 0x5649f135faa0 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T01:04:11.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.statistics: (nil) 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.use_fsync: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_log_file_size: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.allow_fallocate: 1 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.use_direct_reads: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T01:04:11.899 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.db_log_dir: 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.wal_dir: 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.write_buffer_manager: 0x5649f13321e0 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T01:04:11.900 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.unordered_write: 0 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T01:04:11.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T01:04:11.902 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.row_cache: None 2024-04-22T01:04:11.902 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.wal_filter: None 2024-04-22T01:04:11.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T01:04:11.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T01:04:11.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.two_write_queues: 0 2024-04-22T01:04:11.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T01:04:11.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.wal_compression: 0 2024-04-22T01:04:11.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.atomic_flush: 0 2024-04-22T01:04:11.906 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.log_readahead_size: 0 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T01:04:11.907 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_background_jobs: 2 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_background_compactions: -1 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_subcompactions: 1 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_open_files: -1 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T01:04:11.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_background_flushes: -1 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Compression algorithms supported: 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kZSTD supported: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kXpressCompression supported: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kZlibCompression supported: 1 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kSnappyCompression supported: 1 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kLZ4Compression supported: 1 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: kBZip2Compression supported: 0 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T01:04:11.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.merge_operator: 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_filter: None 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5649f135fba0) 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-04-22T01:04:11.910 INFO:journalctl@ceph.mon.b.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: index_type: 0 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: data_block_index_type: 0 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: index_shortening: 1 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: checksum: 4 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: no_block_cache: 0 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache: 0x5649f1363090 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache_options: 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: capacity : 536870912 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: num_shard_bits : 4 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: strict_capacity_limit : 0 2024-04-22T01:04:11.911 INFO:journalctl@ceph.mon.b.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache_compressed: (nil) 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: persistent_cache: (nil) 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_size: 4096 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_size_deviation: 10 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_restart_interval: 16 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: index_block_restart_interval: 1 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: metadata_block_size: 4096 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: partition_filters: 0 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: use_delta_encoding: 1 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: filter_policy: bloomfilter 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: whole_key_filtering: 1 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: verify_compression: 0 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-04-22T01:04:11.912 INFO:journalctl@ceph.mon.b.smithi111.stdout: format_version: 5 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout: enable_index_compression: 1 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_align: 0 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout: max_auto_readahead_size: 262144 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout: prepopulate_block_cache: 0 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout: initial_auto_readahead_size: 8192 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression: NoCompression 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.num_levels: 7 2024-04-22T01:04:11.913 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T01:04:11.914 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T01:04:11.915 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T01:04:11.916 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.table_properties_collectors: 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.inplace_update_support: 0 2024-04-22T01:04:11.917 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.bloom_locality: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.max_successive_merges: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.ttl: 2592000 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T01:04:11.918 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.enable_blob_files: false 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.min_blob_size: 0 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 174f2dea-726b-4fe8-8625-75186d889f5d 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713747851575453, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T01:04:11.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713747851576019, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713747851, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "174f2dea-726b-4fe8-8625-75186d889f5d", "db_session_id": "D3PIQRAS3QGEP39Z5WNG", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713747851576124, "job": 1, "event": "recovery_finished"} 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5649f1454000 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: DB pointer 0x5649f143e000 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** DB Stats ** 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T01:04:11.920 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** Compaction Stats [default] ** 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** Compaction Stats [default] ** 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T01:04:11.921 INFO:journalctl@ceph.mon.b.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Block cache BinnedLRUCache@0x5649f1363090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-22T01:04:11.922 INFO:journalctl@ceph.mon.b.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: using public_addr v2:172.21.15.111:0/0 -> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: starting mon.b rank -1 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(???) e0 preinit fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).mds e1 new map 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: e1 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: legacy client fscid: -1 2024-04-22T01:04:11.923 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout: No filesystems configured 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:04:11.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Deploying cephadm binary to smithi203 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Added host smithi203 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:04:11.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/224248922' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/224248922' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.927 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c;count:3 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:11.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Deploying daemon mon.c on smithi203 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.a calling monitor election 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.c calling monitor election 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.929 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: monmap epoch 2 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: last_changed 2024-04-22T01:04:02.635331+0000 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: min_mon_release 19 (squid) 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: election_strategy: 1 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: fsmap 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mgrmap e13: x(active, since 51s) 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: overall HEALTH_OK 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.930 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: Deploying daemon mon.b on smithi111 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/1896725288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 ceph-mon[31057]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 bash[31038]: 1e22d75ebb5d468af3bfa99119864d4424adbdb1150ddb8f45064fb2045b17e9 2024-04-22T01:04:11.931 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:11 smithi111 systemd[1]: Started Ceph mon.b for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:04:17.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: mon.a calling monitor election 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: mon.c calling monitor election 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: monmap epoch 3 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: last_changed 2024-04-22T01:04:11.604142+0000 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: min_mon_release 19 (squid) 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: election_strategy: 1 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: fsmap 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:17.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: mgrmap e13: x(active, since 60s) 2024-04-22T01:04:17.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: overall HEALTH_OK 2024-04-22T01:04:17.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 -- 172.21.15.203:0/3281482872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 msgr2=0x7f5de4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:17.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3281482872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5de4005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc100b50 tx=0x7f5df4009e80 comp rx=0 tx=0).stop 2024-04-22T01:04:17.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 -- 172.21.15.203:0/3281482872 shutdown_connections 2024-04-22T01:04:17.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3281482872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5de4005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 -- 172.21.15.203:0/3281482872 >> 172.21.15.203:0/3281482872 conn(0x7f5dfc0fc710 msgr2=0x7f5dfc0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:17.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 -- 172.21.15.203:0/3281482872 shutdown_connections 2024-04-22T01:04:17.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.059+0000 7f5e02d02700 1 -- 172.21.15.203:0/3281482872 wait complete. 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 Processor -- start 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 -- start start 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5dfc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 0x7f5dfc198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc108c60 con 0x7f5de40035c0 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc108ae0 con 0x7f5dfc197ce0 2024-04-22T01:04:17.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.060+0000 7f5e02d02700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc108de0 con 0x7f5dfc19db60 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51724/0 (socket says 172.21.15.203:51724) 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 -- 172.21.15.203:0/3094444793 learned_addr learned my addr 172.21.15.203:0/3094444793 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e0129f700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 0x7f5dfc198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5dfc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 msgr2=0x7f5dfc1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 msgr2=0x7f5dfc1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5dfbfff700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e0129f700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 msgr2=0x7f5dfc198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e0129f700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 msgr2=0x7f5dfc198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e0129f700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 0x7f5dfc198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T01:04:17.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e0129f700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 0x7f5dfc198190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 msgr2=0x7f5dfc1977a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 msgr2=0x7f5dfc198190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 0x7f5dfc198190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 -- 172.21.15.203:0/3094444793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc1a4480 con 0x7f5de40035c0 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e00a9e700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5dfc19d620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5df4002230 tx=0x7f5df40048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df401a730 con 0x7f5de40035c0 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc1a46b0 con 0x7f5de40035c0 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5df4026030 con 0x7f5de40035c0 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.061+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df40058b0 con 0x7f5de40035c0 2024-04-22T01:04:17.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.062+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc1a4bc0 con 0x7f5de40035c0 2024-04-22T01:04:17.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.063+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f5df4015040 con 0x7f5de40035c0 2024-04-22T01:04:17.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.063+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dfc102360 con 0x7f5de40035c0 2024-04-22T01:04:17.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.063+0000 7f5df9ffb700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5de8039580 0x7f5de803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:17.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.063+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5df404e7e0 con 0x7f5de40035c0 2024-04-22T01:04:17.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.063+0000 7f5dfbfff700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5de8039580 0x7f5de803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:17.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.063+0000 7f5dfbfff700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5de8039580 0x7f5de803ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009180 tx=0x7f5df0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:17.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.068+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5df4015330 con 0x7f5de40035c0 2024-04-22T01:04:17.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:04:17.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: mon.a calling monitor election 2024-04-22T01:04:17.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: mon.c calling monitor election 2024-04-22T01:04:17.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:17.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: monmap epoch 3 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: last_changed 2024-04-22T01:04:11.604142+0000 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: min_mon_release 19 (squid) 2024-04-22T01:04:17.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: election_strategy: 1 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: fsmap 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: mgrmap e13: x(active, since 60s) 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: overall HEALTH_OK 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:17.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.359+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5dfc02db30 con 0x7f5de40035c0 2024-04-22T01:04:17.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.360+0000 7f5df9ffb700 1 -- 172.21.15.203:0/3094444793 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f5df4005050 con 0x7f5de40035c0 2024-04-22T01:04:17.360 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-22T01:04:17.360 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","modified":"2024-04-22T01:04:11.604142Z","created":"2024-04-22T01:02:16.910765Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T01:04:17.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5de8039580 msgr2=0x7f5de803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:17.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5de8039580 0x7f5de803ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009180 tx=0x7f5df0008040 comp rx=0 tx=0).stop 2024-04-22T01:04:17.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 msgr2=0x7f5dfc19d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:17.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5dfc19d620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5df4002230 tx=0x7f5df40048c0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.362 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 shutdown_connections 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5de8039580 0x7f5de803ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dfc197ce0 0x7f5dfc198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5de40035c0 0x7f5dfc19d620 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 --2- 172.21.15.203:0/3094444793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5dfc19db60 0x7f5dfc1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 >> 172.21.15.203:0/3094444793 conn(0x7f5dfc0fc710 msgr2=0x7f5dfc06a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 shutdown_connections 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:17.362+0000 7f5e02d02700 1 -- 172.21.15.203:0/3094444793 wait complete. 2024-04-22T01:04:17.363 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-04-22T01:04:17.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:17.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:17 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/3094444793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T01:04:17.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:17.975 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T01:04:17.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T01:04:18.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:18.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:17 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/3094444793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T01:04:18.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.232 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:04:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:04:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: mon.a calling monitor election 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: mon.c calling monitor election 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: monmap epoch 3 2024-04-22T01:04:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: last_changed 2024-04-22T01:04:11.604142+0000 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: min_mon_release 19 (squid) 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: election_strategy: 1 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: fsmap 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: mgrmap e13: x(active, since 60s) 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: overall HEALTH_OK 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:18.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:18.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:18.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/3094444793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T01:04:18.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: mon.b calling monitor election 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: mon.b calling monitor election 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: mon.a calling monitor election 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: mon.c calling monitor election 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: monmap epoch 3 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: last_changed 2024-04-22T01:04:11.604142+0000 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: min_mon_release 19 (squid) 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: election_strategy: 1 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: fsmap 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: mgrmap e13: x(active, since 62s) 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: overall HEALTH_OK 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: mon.b calling monitor election 2024-04-22T01:04:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: mon.b calling monitor election 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: mon.a calling monitor election 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: mon.c calling monitor election 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: monmap epoch 3 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: last_changed 2024-04-22T01:04:11.604142+0000 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: min_mon_release 19 (squid) 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: election_strategy: 1 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: fsmap 2024-04-22T01:04:19.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:19.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: mgrmap e13: x(active, since 62s) 2024-04-22T01:04:19.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: overall HEALTH_OK 2024-04-22T01:04:19.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:19.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:19.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:19.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:19 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: mon.b calling monitor election 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: mon.b calling monitor election 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: mon.a calling monitor election 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: mon.c calling monitor election 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: monmap epoch 3 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: last_changed 2024-04-22T01:04:11.604142+0000 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: created 2024-04-22T01:02:16.910765+0000 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: min_mon_release 19 (squid) 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: election_strategy: 1 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: fsmap 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: mgrmap e13: x(active, since 62s) 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: overall HEALTH_OK 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:19.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:19 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:20.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 -- 172.21.15.38:0/2963535590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 msgr2=0x7f0f28102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:20.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 --2- 172.21.15.38:0/2963535590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f28102280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009910 tx=0x7f0f18009c20 comp rx=0 tx=0).stop 2024-04-22T01:04:20.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 -- 172.21.15.38:0/2963535590 shutdown_connections 2024-04-22T01:04:20.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 --2- 172.21.15.38:0/2963535590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f28102280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 -- 172.21.15.38:0/2963535590 >> 172.21.15.38:0/2963535590 conn(0x7f0f280fd770 msgr2=0x7f0f280ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:20.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 -- 172.21.15.38:0/2963535590 shutdown_connections 2024-04-22T01:04:20.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 -- 172.21.15.38:0/2963535590 wait complete. 2024-04-22T01:04:20.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.060+0000 7f0f2d851700 1 Processor -- start 2024-04-22T01:04:20.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 -- start start 2024-04-22T01:04:20.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f2819d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:20.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f2819dbd0 0x7f0f28197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f28197d50 0x7f0f28198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2810c230 con 0x7f0f28101ea0 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2810c0b0 con 0x7f0f28197d50 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f2d851700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2810c3b0 con 0x7f0f2819dbd0 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f2819d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f2819d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44606/0 (socket says 172.21.15.38:44606) 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 -- 172.21.15.38:0/1733421425 learned_addr learned my addr 172.21.15.38:0/1733421425 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 -- 172.21.15.38:0/1733421425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f2819dbd0 msgr2=0x7f0f28197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f2819dbd0 0x7f0f28197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 -- 172.21.15.38:0/1733421425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f28197d50 msgr2=0x7f0f28198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f277fe700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f28197d50 0x7f0f28198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f28197d50 0x7f0f28198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f26ffd700 1 -- 172.21.15.38:0/1733421425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f281a44f0 con 0x7f0f28101ea0 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.061+0000 7f0f277fe700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f28197d50 0x7f0f28198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.062+0000 7f0f26ffd700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f2819d690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800bdc0 tx=0x7f0f1800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:20.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.062+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18026070 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.062+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f180050d0 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.062+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f281a4780 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.062+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1800fdf0 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.062+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f281a4c90 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.063+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0f18029070 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.063+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f28103380 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.063+0000 7f0f2c84f700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f10039580 0x7f0f1003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.063+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0f1804cd80 con 0x7f0f28101ea0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.063+0000 7f0f267fc700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f10039580 0x7f0f1003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:20.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.063+0000 7f0f267fc700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f10039580 0x7f0f1003ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0098e0 tx=0x7f0f1c006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:20.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.068+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0f18013070 con 0x7f0f28101ea0 2024-04-22T01:04:20.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.280+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f0f281a4ef0 con 0x7f0f28101ea0 2024-04-22T01:04:20.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.280+0000 7f0f2c84f700 1 -- 172.21.15.38:0/1733421425 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f0f18030020 con 0x7f0f28101ea0 2024-04-22T01:04:20.281 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:20.281 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-04-22T01:04:20.281 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:04:20.281 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-22T01:04:20.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f10039580 msgr2=0x7f0f1003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:20.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f10039580 0x7f0f1003ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0098e0 tx=0x7f0f1c006dd0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 msgr2=0x7f0f2819d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f2819d690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800bdc0 tx=0x7f0f1800bea0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 shutdown_connections 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f10039580 0x7f0f1003ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f28197d50 0x7f0f28198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f28101ea0 0x7f0f2819d690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 --2- 172.21.15.38:0/1733421425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f2819dbd0 0x7f0f28197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 >> 172.21.15.38:0/1733421425 conn(0x7f0f280fd770 msgr2=0x7f0f28103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 shutdown_connections 2024-04-22T01:04:20.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:20.282+0000 7f0f2d851700 1 -- 172.21.15.38:0/1733421425 wait complete. 2024-04-22T01:04:20.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T01:04:20.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T01:04:20.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T01:04:20.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Updating smithi038:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:20.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:20.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:20.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: Reconfiguring daemon mon.a on smithi038 2024-04-22T01:04:20.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Updating smithi038:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: Reconfiguring daemon mon.a on smithi038 2024-04-22T01:04:20.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:20.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Updating smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Updating smithi038:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Updating smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: Reconfiguring daemon mon.a on smithi038 2024-04-22T01:04:20.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:04:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:21 smithi203 ceph-mon[31825]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:21 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1733421425' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:21.553 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:21 smithi038 ceph-mon[28043]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:21.553 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:21 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1733421425' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:21.554 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:21 smithi111 ceph-mon[31057]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:21 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1733421425' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:21.717 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T01:04:21.717 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:04:21.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T01:04:21.785 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:04:21.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:04:21.850 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:04:21.885 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T01:04:21.911 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:04:21.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:04:21.974 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:04:21.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T01:04:22.005 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:04:22.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T01:04:22.074 INFO:tasks.cephadm:Adding mgr.x on smithi038 2024-04-22T01:04:22.074 INFO:tasks.cephadm:Adding mgr.y on smithi111 2024-04-22T01:04:22.074 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-04-22T01:04:22.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi038=x;smithi111=y;smithi203=z' 2024-04-22T01:04:22.349 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: Reconfiguring mon.b (monmap changed)... 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:22.607 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:22 smithi203 ceph-mon[31825]: Reconfiguring daemon mon.b on smithi111 2024-04-22T01:04:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: Reconfiguring mon.b (monmap changed)... 2024-04-22T01:04:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:22.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:22 smithi111 ceph-mon[31057]: Reconfiguring daemon mon.b on smithi111 2024-04-22T01:04:22.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: Reconfiguring mon.b (monmap changed)... 2024-04-22T01:04:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:22 smithi038 ceph-mon[28043]: Reconfiguring daemon mon.b on smithi111 2024-04-22T01:04:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:23 smithi203 ceph-mon[31825]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:23 smithi111 ceph-mon[31057]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:23.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:23 smithi038 ceph-mon[28043]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:23.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:23.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:23.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T01:04:23.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T01:04:23.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:24.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:24 smithi203 ceph-mon[31825]: Reconfiguring mon.c (monmap changed)... 2024-04-22T01:04:24.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:24 smithi203 ceph-mon[31825]: Reconfiguring daemon mon.c on smithi203 2024-04-22T01:04:24.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:24.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:24.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:24 smithi111 ceph-mon[31057]: Reconfiguring mon.c (monmap changed)... 2024-04-22T01:04:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:24 smithi111 ceph-mon[31057]: Reconfiguring daemon mon.c on smithi203 2024-04-22T01:04:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:24.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:24.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:24 smithi038 ceph-mon[28043]: Reconfiguring mon.c (monmap changed)... 2024-04-22T01:04:24.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:24 smithi038 ceph-mon[28043]: Reconfiguring daemon mon.c on smithi203 2024-04-22T01:04:24.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:24.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:24.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 -- 172.21.15.203:0/3664084608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103e20 msgr2=0x7f762010a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:24.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 --2- 172.21.15.203:0/3664084608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103e20 0x7f762010a6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7610009990 tx=0x7f7610009ca0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 -- 172.21.15.203:0/3664084608 shutdown_connections 2024-04-22T01:04:24.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 --2- 172.21.15.203:0/3664084608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103e20 0x7f762010a6d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 --2- 172.21.15.203:0/3664084608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620102280 0x7f76201026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 --2- 172.21.15.203:0/3664084608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7620103480 0x7f76201038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.655+0000 7f7627c4f700 1 -- 172.21.15.203:0/3664084608 >> 172.21.15.203:0/3664084608 conn(0x7f76200fd820 msgr2=0x7f76200ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:24.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 -- 172.21.15.203:0/3664084608 shutdown_connections 2024-04-22T01:04:24.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 -- 172.21.15.203:0/3664084608 wait complete. 2024-04-22T01:04:24.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 Processor -- start 2024-04-22T01:04:24.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 -- start start 2024-04-22T01:04:24.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7620102280 0x7f7620198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:24.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 0x7f7620198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:24.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620103e20 0x7f762019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:24.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f762010de60 con 0x7f7620103e20 2024-04-22T01:04:24.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f762010dce0 con 0x7f7620102280 2024-04-22T01:04:24.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f7627c4f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f762010dfe0 con 0x7f7620103480 2024-04-22T01:04:24.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f76251ea700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 0x7f7620198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f76251ea700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 0x7f7620198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:50872/0 (socket says 172.21.15.203:50872) 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f76251ea700 1 -- 172.21.15.203:0/475987176 learned_addr learned my addr 172.21.15.203:0/475987176 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.656+0000 7f76261ec700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620103e20 0x7f762019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76259eb700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7620102280 0x7f7620198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76251ea700 1 -- 172.21.15.203:0/475987176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7620102280 msgr2=0x7f7620198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76251ea700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7620102280 0x7f7620198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76251ea700 1 -- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620103e20 msgr2=0x7f762019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:24.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76251ea700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620103e20 0x7f762019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76251ea700 1 -- 172.21.15.203:0/475987176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76201a00a0 con 0x7f7620103480 2024-04-22T01:04:24.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76261ec700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620103e20 0x7f762019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:04:24.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f76251ea700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 0x7f7620198980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f761400d9b0 tx=0x7f761400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:24.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614017070 con 0x7f7620103480 2024-04-22T01:04:24.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76201a0390 con 0x7f7620103480 2024-04-22T01:04:24.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7614005020 con 0x7f7620103480 2024-04-22T01:04:24.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614009bd0 con 0x7f7620103480 2024-04-22T01:04:24.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.657+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76201a0880 con 0x7f7620103480 2024-04-22T01:04:24.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.658+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7614010460 con 0x7f7620103480 2024-04-22T01:04:24.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.658+0000 7f761effd700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7608039330 0x7f760803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:24.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.658+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f761404c290 con 0x7f7620103480 2024-04-22T01:04:24.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.658+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7620068f20 con 0x7f7620103480 2024-04-22T01:04:24.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.658+0000 7f76259eb700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7608039330 0x7f760803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:24.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.659+0000 7f76259eb700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7608039330 0x7f760803b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76201032e0 tx=0x7f761800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:24.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.663+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7614020020 con 0x7f7620103480 2024-04-22T01:04:24.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.870+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}) -- 0x7f7620062670 con 0x7f7608039330 2024-04-22T01:04:24.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.883+0000 7f761effd700 1 -- 172.21.15.203:0/475987176 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7620062670 con 0x7f7608039330 2024-04-22T01:04:24.883 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-04-22T01:04:24.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7608039330 msgr2=0x7f760803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:24.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7608039330 0x7f760803b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76201032e0 tx=0x7f761800b040 comp rx=0 tx=0).stop 2024-04-22T01:04:24.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 msgr2=0x7f7620198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:24.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 0x7f7620198980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f761400d9b0 tx=0x7f761400dcc0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 shutdown_connections 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7608039330 0x7f760803b7f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7620103e20 0x7f762019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7620102280 0x7f7620198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 --2- 172.21.15.203:0/475987176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7620103480 0x7f7620198980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 >> 172.21.15.203:0/475987176 conn(0x7f76200fd820 msgr2=0x7f7620106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 shutdown_connections 2024-04-22T01:04:24.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:04:24.885+0000 7f7627c4f700 1 -- 172.21.15.203:0/475987176 wait complete. 2024-04-22T01:04:25.499 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:25.500 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:25.525 DEBUG:teuthology.orchestra.run.smithi111:mgr.y> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T01:04:25.527 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.z.service 2024-04-22T01:04:25.529 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T01:04:25.530 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:04:25.530 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:04:25.549 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:04:25.549 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-22T01:04:25.608 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:04:25.608 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:25.608 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-22T01:04:25.608 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:25.608 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:25.608 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 01:03:28.909649098 +0000 2024-04-22T01:04:25.609 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 00:57:45.620074778 +0000 2024-04-22T01:04:25.609 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 00:57:45.620074778 +0000 2024-04-22T01:04:25.609 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 00:57:45.620074778 +0000 2024-04-22T01:04:25.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:04:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:25.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:25.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:25.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:25.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T01:04:25.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:25.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:25.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:25.677 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T01:04:25.677 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T01:04:25.677 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000323536 s, 1.6 MB/s 2024-04-22T01:04:25.679 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:04:25.738 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 01:03:28.943648484 +0000 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 00:57:45.972068364 +0000 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 00:57:45.972068364 +0000 2024-04-22T01:04:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 00:57:45.972068364 +0000 2024-04-22T01:04:25.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:04:25.860 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T01:04:25.861 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T01:04:25.861 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000294664 s, 1.7 MB/s 2024-04-22T01:04:25.862 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:04:25.919 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-22T01:04:25.976 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:04:25.976 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:25.976 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T01:04:25.977 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:25.977 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:25.977 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 01:03:28.976647889 +0000 2024-04-22T01:04:25.977 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 00:57:46.327061896 +0000 2024-04-22T01:04:25.977 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 00:57:46.327061896 +0000 2024-04-22T01:04:25.977 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 00:57:46.327061896 +0000 2024-04-22T01:04:25.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:04:26.042 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T01:04:26.042 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T01:04:26.042 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000281582 s, 1.8 MB/s 2024-04-22T01:04:26.043 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:04:26.101 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-22T01:04:26.156 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 01:03:29.006647347 +0000 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 00:57:46.687055337 +0000 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 00:57:46.687055337 +0000 2024-04-22T01:04:26.157 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 00:57:46.687055337 +0000 2024-04-22T01:04:26.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:04:26.220 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-22T01:04:26.221 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-22T01:04:26.221 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000364258 s, 1.4 MB/s 2024-04-22T01:04:26.222 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:04:26.278 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-22T01:04:26.279 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:04:26.298 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:04:26.298 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-22 01:04:11.028524996 +0000 2024-04-22T01:04:26.356 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-22 00:57:57.578489635 +0000 2024-04-22T01:04:26.357 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-22 00:57:57.578489635 +0000 2024-04-22T01:04:26.357 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-22 00:57:57.578489635 +0000 2024-04-22T01:04:26.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:04:26.422 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-22T01:04:26.422 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-22T01:04:26.422 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000356186 s, 1.4 MB/s 2024-04-22T01:04:26.424 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:04:26.482 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-22 01:04:11.028524996 +0000 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-22 00:57:57.928483444 +0000 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-22 00:57:57.928483444 +0000 2024-04-22T01:04:26.540 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-22 00:57:57.928483444 +0000 2024-04-22T01:04:26.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:04:26.546 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:26 smithi203 ceph-mon[31825]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:26.546 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:26 smithi203 ceph-mon[31825]: Saving service mgr spec with placement smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-22T01:04:26.547 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:26 smithi203 ceph-mon[31825]: Deploying daemon mgr.z on smithi203 2024-04-22T01:04:26.602 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:26 smithi111 ceph-mon[31057]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:26.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:26 smithi111 ceph-mon[31057]: Saving service mgr spec with placement smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-22T01:04:26.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:26 smithi111 ceph-mon[31057]: Deploying daemon mgr.z on smithi203 2024-04-22T01:04:26.605 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-22T01:04:26.605 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-22T01:04:26.605 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000209407 s, 2.4 MB/s 2024-04-22T01:04:26.607 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:04:26.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:26 smithi038 ceph-mon[28043]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:26.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:26 smithi038 ceph-mon[28043]: Saving service mgr spec with placement smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-22T01:04:26.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:26 smithi038 ceph-mon[28043]: Deploying daemon mgr.z on smithi203 2024-04-22T01:04:26.665 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-22 01:04:11.028524996 +0000 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-22 00:57:58.357475855 +0000 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-22 00:57:58.357475855 +0000 2024-04-22T01:04:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-22 00:57:58.357475855 +0000 2024-04-22T01:04:26.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:04:26.786 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-22T01:04:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-22T01:04:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000176009 s, 2.9 MB/s 2024-04-22T01:04:26.788 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:04:26.844 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-04-22T01:04:26.900 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:04:26.900 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-22 01:04:11.029524978 +0000 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-22 00:57:58.698469824 +0000 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-22 00:57:58.698469824 +0000 2024-04-22T01:04:26.901 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-22 00:57:58.698469824 +0000 2024-04-22T01:04:26.901 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:04:26.966 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-22T01:04:26.966 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-22T01:04:26.966 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000174259 s, 2.9 MB/s 2024-04-22T01:04:26.968 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:04:27.025 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-22T01:04:27.025 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:04:27.043 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:04:27.043 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-22T01:04:27.098 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:04:27.098 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:27.098 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T01:04:27.099 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:27.099 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:27.099 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-22 01:04:26.824456366 +0000 2024-04-22T01:04:27.099 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-22 00:58:20.363292470 +0000 2024-04-22T01:04:27.099 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-22 00:58:20.363292470 +0000 2024-04-22T01:04:27.099 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-22 00:58:20.363292470 +0000 2024-04-22T01:04:27.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:04:27.160 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 podman[33458]: 2024-04-22 01:04:27.134685291 +0000 UTC m=+0.144599829 container create 3277d117afd75cb0e8bfabbd1d667ab1f59c44e9a76fbf7ead5d90cef22f7b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T01:04:27.163 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-22T01:04:27.163 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-22T01:04:27.163 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000268112 s, 1.9 MB/s 2024-04-22T01:04:27.164 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:04:27.221 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-22T01:04:27.278 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-22 01:04:26.825456341 +0000 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-22 00:58:20.694284288 +0000 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-22 00:58:20.694284288 +0000 2024-04-22T01:04:27.279 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-22 00:58:20.694284288 +0000 2024-04-22T01:04:27.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:04:27.341 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-22T01:04:27.341 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-22T01:04:27.341 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000177509 s, 2.9 MB/s 2024-04-22T01:04:27.342 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:04:27.399 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-22 01:04:26.825456341 +0000 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-22 00:58:21.034275884 +0000 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-22 00:58:21.034275884 +0000 2024-04-22T01:04:27.454 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-22 00:58:21.034275884 +0000 2024-04-22T01:04:27.455 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:04:27.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:27 smithi203 ceph-mon[31825]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:27.470 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 podman[33458]: 2024-04-22 01:04:27.370929821 +0000 UTC m=+0.380844363 container init 3277d117afd75cb0e8bfabbd1d667ab1f59c44e9a76fbf7ead5d90cef22f7b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T01:04:27.470 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 podman[33458]: 2024-04-22 01:04:27.37387422 +0000 UTC m=+0.383788765 container start 3277d117afd75cb0e8bfabbd1d667ab1f59c44e9a76fbf7ead5d90cef22f7b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T01:04:27.471 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 ceph-mgr[33580]: -- 172.21.15.203:0/3904017306 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da224ee5a0 con 0x55da224c3000 2024-04-22T01:04:27.494 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-22T01:04:27.495 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-22T01:04:27.495 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000215865 s, 2.4 MB/s 2024-04-22T01:04:27.496 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:04:27.552 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-22T01:04:27.607 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-22 01:04:26.825456341 +0000 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-22 00:58:21.377267407 +0000 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-22 00:58:21.377267407 +0000 2024-04-22T01:04:27.608 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-22 00:58:21.377267407 +0000 2024-04-22T01:04:27.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:04:27.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:27 smithi111 ceph-mon[31057]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:27.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:27 smithi038 ceph-mon[28043]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:27.669 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-22T01:04:27.669 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-22T01:04:27.670 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000183093 s, 2.8 MB/s 2024-04-22T01:04:27.671 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:04:27.727 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-22T01:04:27.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T01:04:27.729 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 bash[33458]: 3277d117afd75cb0e8bfabbd1d667ab1f59c44e9a76fbf7ead5d90cef22f7b67 2024-04-22T01:04:27.729 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 systemd[1]: Started Ceph mgr.z for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:04:27.729 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:27.534+0000 7f08454d8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:04:27.730 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:27.729+0000 7f08454d8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:04:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:04:28.015 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:27.834+0000 7f08454d8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:04:28.516 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:28.149+0000 7f08454d8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:04:28.516 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:28.235+0000 7f08454d8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:28.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:28 smithi038 ceph-mon[28043]: Deploying daemon mgr.y on smithi111 2024-04-22T01:04:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T01:04:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:28.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:28.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:28 smithi111 ceph-mon[31057]: Deploying daemon mgr.y on smithi111 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:28 smithi203 ceph-mon[31825]: Deploying daemon mgr.y on smithi111 2024-04-22T01:04:29.759 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:29 smithi203 ceph-mon[31825]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:29.759 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:29.424+0000 7f08454d8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:04:29.759 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:29.589+0000 7f08454d8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:04:29.759 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:29.759+0000 7f08454d8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:04:29.844 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:29 smithi111 ceph-mon[31057]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:29.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:29 smithi038 ceph-mon[28043]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:29.911 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T01:04:29.912 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T01:04:29.912 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T01:04:29.912 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T01:04:29.912 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00724424 s, 1.4 GB/s 2024-04-22T01:04:29.912 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T01:04:30.354 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:30.040+0000 7f08454d8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:04:30.354 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:30.132+0000 7f08454d8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:04:30.355 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:30.354+0000 7f08454d8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:04:30.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-22T01:04:30.766 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:30.554+0000 7f08454d8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:04:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:04:31.096 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:30 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:30.833+0000 7f0d248dc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:04:31.272 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:31.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:31.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:31.191+0000 7f08454d8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:31.929 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:31.631+0000 7f08454d8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:04:31.929 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:31.728+0000 7f08454d8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:04:31.929 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:31.929+0000 7f08454d8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:04:32.395 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:32.066+0000 7f0d248dc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:04:32.395 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:32.232+0000 7f0d248dc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:32.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:32.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:32.687 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:32.402+0000 7f0d248dc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:04:32.687 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:32.687+0000 7f0d248dc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:04:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:32.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:32.457+0000 7f08454d8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:32.547+0000 7f08454d8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:32.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:32.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:32.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T01:04:32.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T01:04:32.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 -- 172.21.15.38:0/262867013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf741023b0 msgr2=0x7faf74102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 --2- 172.21.15.38:0/262867013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf741023b0 0x7faf74102810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faf680099c0 tx=0x7faf68009cd0 comp rx=0 tx=0).stop 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 -- 172.21.15.38:0/262867013 shutdown_connections 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 --2- 172.21.15.38:0/262867013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf74102d50 0x7faf74109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 --2- 172.21.15.38:0/262867013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 0x7faf741015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 --2- 172.21.15.38:0/262867013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf741023b0 0x7faf74102810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 -- 172.21.15.38:0/262867013 >> 172.21.15.38:0/262867013 conn(0x7faf740fc750 msgr2=0x7faf740feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 -- 172.21.15.38:0/262867013 shutdown_connections 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.790+0000 7faf79d70700 1 -- 172.21.15.38:0/262867013 wait complete. 2024-04-22T01:04:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 Processor -- start 2024-04-22T01:04:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 -- start start 2024-04-22T01:04:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 0x7faf74198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf741023b0 0x7faf74198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 0x7faf7419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faf7410cde0 con 0x7faf74102d50 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faf7410cc60 con 0x7faf741023b0 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf79d70700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faf7410cf60 con 0x7faf741011b0 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 0x7faf7419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf737fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 0x7faf74198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 0x7faf7419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41220/0 (socket says 172.21.15.38:41220) 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 -- 172.21.15.38:0/1017556087 learned_addr learned my addr 172.21.15.38:0/1017556087 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf72ffd700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf741023b0 0x7faf74198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 -- 172.21.15.38:0/1017556087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 msgr2=0x7faf74198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 0x7faf74198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 -- 172.21.15.38:0/1017556087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf741023b0 msgr2=0x7faf74198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf741023b0 0x7faf74198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf73fff700 1 -- 172.21.15.38:0/1017556087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf741a0170 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.791+0000 7faf737fe700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 0x7faf74198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.792+0000 7faf73fff700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 0x7faf7419fad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf6c007f10 tx=0x7faf6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.792+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c00dd20 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.792+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf6c004020 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.792+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c015660 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.792+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf741a0460 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.792+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf741a09a0 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.793+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7faf6c003a00 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.793+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf74068f20 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.793+0000 7faf70ff9700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faf5c039330 0x7faf5c03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.793+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7faf6c04bcf0 con 0x7faf74102d50 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.793+0000 7faf737fe700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faf5c039330 0x7faf5c03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.794+0000 7faf737fe700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faf5c039330 0x7faf5c03b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf74102210 tx=0x7faf64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:32.798+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7faf6c010070 con 0x7faf74102d50 2024-04-22T01:04:32.978 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:32.779+0000 7f0d248dc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:04:32.979 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:32.979+0000 7f0d248dc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:04:33.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:33.015+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7faf740fe960 con 0x7faf5c039330 2024-04-22T01:04:33.132 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:32.788+0000 7f08454d8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:04:33.132 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:32.922+0000 7f08454d8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:04:33.132 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:33.017+0000 7f08454d8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:04:33.132 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:33.132+0000 7f08454d8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:04:33.395 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:33.170+0000 7f0d248dc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:04:33.516 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:33.222+0000 7f08454d8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:04:33.728 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:33 smithi038 ceph-mon[28043]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:33.728 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:33 smithi038 ceph-mon[28043]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T01:04:33.728 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:33 smithi038 ceph-mon[28043]: Reconfiguring daemon mgr.x on smithi038 2024-04-22T01:04:33.728 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:04:33.728 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:04:33.728 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:33.895 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:33.761+0000 7f0d248dc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:04:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-mon[31057]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-mon[31057]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T01:04:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-mon[31057]: Reconfiguring daemon mgr.x on smithi038 2024-04-22T01:04:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:04:33.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:04:33.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-mon[31825]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-mon[31825]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T01:04:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-mon[31825]: Reconfiguring daemon mgr.x on smithi038 2024-04-22T01:04:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:04:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:04:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:34.472 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:34.181+0000 7f0d248dc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:04:34.473 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:34.276+0000 7f0d248dc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:04:34.473 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:34.473+0000 7f0d248dc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:04:34.516 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:34.057+0000 7f08454d8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:04:34.516 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z[33542]: 2024-04-22T01:04:34.270+0000 7f08454d8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:04:34.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:34.845+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7faf6c012350 con 0x7faf74102d50 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: Standby manager daemon z started 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T01:04:34.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T01:04:34.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:34 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: Standby manager daemon z started 2024-04-22T01:04:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T01:04:34.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T01:04:34.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T01:04:34.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: Standby manager daemon z started 2024-04-22T01:04:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T01:04:35.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T01:04:35.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T01:04:35.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:34 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.203:0/2288956387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T01:04:35.317 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:34.992+0000 7f0d248dc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:04:35.317 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:35.082+0000 7f0d248dc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:04:35.645 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:35.317+0000 7f0d248dc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:04:35.645 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:35.450+0000 7f0d248dc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:04:35.645 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:35.544+0000 7f0d248dc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:04:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:35 smithi038 ceph-mon[28043]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:35 smithi038 ceph-mon[28043]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-22T01:04:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:35 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T01:04:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:35 smithi203 ceph-mon[31825]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:35 smithi203 ceph-mon[31825]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-22T01:04:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:35 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T01:04:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-mon[31057]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-mon[31057]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-22T01:04:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T01:04:36.145 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:35.657+0000 7f0d248dc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:04:36.145 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:35.747+0000 7f0d248dc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:04:36.895 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:36.550+0000 7f0d248dc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:04:36.895 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y[32430]: 2024-04-22T01:04:36.762+0000 7f0d248dc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:04:36.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-mon[31057]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:36.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2353570077' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c263cffd-6624-47e2-935b-671cc5cb79b2"}]: dispatch 2024-04-22T01:04:36.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2353570077' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c263cffd-6624-47e2-935b-671cc5cb79b2"}]': finished 2024-04-22T01:04:36.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-mon[31057]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T01:04:36.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:36 smithi038 ceph-mon[28043]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2353570077' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c263cffd-6624-47e2-935b-671cc5cb79b2"}]: dispatch 2024-04-22T01:04:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2353570077' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c263cffd-6624-47e2-935b-671cc5cb79b2"}]': finished 2024-04-22T01:04:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:36 smithi038 ceph-mon[28043]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T01:04:36.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:36 smithi203 ceph-mon[31825]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2353570077' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c263cffd-6624-47e2-935b-671cc5cb79b2"}]: dispatch 2024-04-22T01:04:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2353570077' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c263cffd-6624-47e2-935b-671cc5cb79b2"}]': finished 2024-04-22T01:04:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:36 smithi203 ceph-mon[31825]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T01:04:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:37.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:37.659+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7faf6c013b40 con 0x7faf74102d50 2024-04-22T01:04:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:37 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3116007554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:04:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:37 smithi203 ceph-mon[31825]: Standby manager daemon y started 2024-04-22T01:04:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:37 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T01:04:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:37 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T01:04:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:37 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T01:04:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:37 smithi203 ceph-mon[31825]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T01:04:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:37 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3116007554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:04:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:37 smithi111 ceph-mon[31057]: Standby manager daemon y started 2024-04-22T01:04:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:37 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T01:04:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:37 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T01:04:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:37 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T01:04:38.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:37 smithi111 ceph-mon[31057]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T01:04:38.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:37 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3116007554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:04:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:37 smithi038 ceph-mon[28043]: Standby manager daemon y started 2024-04-22T01:04:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:37 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T01:04:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:37 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T01:04:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:37 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T01:04:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:37 smithi038 ceph-mon[28043]: from='mgr.? 172.21.15.111:0/2626170518' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T01:04:39.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:38 smithi203 ceph-mon[31825]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-22T01:04:39.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:38 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T01:04:39.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:38 smithi203 ceph-mon[31825]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:39.055 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:38 smithi038 ceph-mon[28043]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-22T01:04:39.055 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:38 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T01:04:39.055 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:38 smithi038 ceph-mon[28043]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:39.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:38 smithi111 ceph-mon[31057]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-22T01:04:39.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:38 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T01:04:39.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:38 smithi111 ceph-mon[31057]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:41.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:40 smithi038 ceph-mon[28043]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:41.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:40 smithi111 ceph-mon[31057]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:41.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:40 smithi203 ceph-mon[31825]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:43.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:42 smithi038 ceph-mon[28043]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:43.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:42 smithi111 ceph-mon[31057]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:43.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:42 smithi203 ceph-mon[31825]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:44.126 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T01:04:44.126 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:44.126 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:43 smithi038 ceph-mon[28043]: Deploying daemon osd.0 on smithi038 2024-04-22T01:04:44.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T01:04:44.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:44.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:43 smithi111 ceph-mon[31057]: Deploying daemon osd.0 on smithi038 2024-04-22T01:04:44.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T01:04:44.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:44.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:43 smithi203 ceph-mon[31825]: Deploying daemon osd.0 on smithi038 2024-04-22T01:04:45.134 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:44 smithi038 ceph-mon[28043]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:45.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:44 smithi111 ceph-mon[31057]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:45.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:44 smithi203 ceph-mon[31825]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:47.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:46 smithi038 ceph-mon[28043]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:47.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:46 smithi111 ceph-mon[31057]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:47.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:46 smithi203 ceph-mon[31825]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:48.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:48 smithi038 ceph-mon[28043]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:49.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:48 smithi111 ceph-mon[31057]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:48 smithi203 ceph-mon[31825]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:50.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:50.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:50.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:50.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:51.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:51 smithi038 ceph-mon[28043]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:51.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:51 smithi203 ceph-mon[31825]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:51.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:51 smithi111 ceph-mon[31057]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:52.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:52.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:52.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:52.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:52.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:52 smithi038 ceph-mon[28043]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:52.804 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2024-04-22T01:04:52.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.804+0000 7faf70ff9700 1 -- 172.21.15.38:0/1017556087 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faf740fe960 con 0x7faf5c039330 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faf5c039330 msgr2=0x7faf5c03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faf5c039330 0x7faf5c03b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf74102210 tx=0x7faf64009260 comp rx=0 tx=0).stop 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 msgr2=0x7faf7419fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 0x7faf7419fad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf6c007f10 tx=0x7faf6c00d3b0 comp rx=0 tx=0).stop 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 shutdown_connections 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faf5c039330 0x7faf5c03b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf74102d50 0x7faf7419fad0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf741011b0 0x7faf74198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 --2- 172.21.15.38:0/1017556087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf741023b0 0x7faf74198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.806+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 >> 172.21.15.38:0/1017556087 conn(0x7faf740fc750 msgr2=0x7faf740fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:52.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.807+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 shutdown_connections 2024-04-22T01:04:52.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:04:52.807+0000 7faf79d70700 1 -- 172.21.15.38:0/1017556087 wait complete. 2024-04-22T01:04:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:52 smithi203 ceph-mon[31825]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:52 smithi111 ceph-mon[31057]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:53.472 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T01:04:53.475 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_4... 2024-04-22T01:04:53.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T01:04:53.719 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:04:54.046 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:53 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:54.046 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:53 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:54.046 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:53 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:54.047 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:53 smithi111 ceph-mon[31057]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T01:04:54.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:54.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:54.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:54.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:53 smithi038 ceph-mon[28043]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T01:04:54.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:53 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:54.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:53 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:54.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:53 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:04:54.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:53 smithi203 ceph-mon[31825]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T01:04:55.032 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:54 smithi111 ceph-mon[31057]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:55.032 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:54 smithi111 ceph-mon[31057]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T01:04:55.032 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:54 smithi111 ceph-mon[31057]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T01:04:55.033 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:54 smithi111 ceph-mon[31057]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:04:55.033 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:55.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-mon[28043]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:55.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-mon[28043]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T01:04:55.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-mon[28043]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T01:04:55.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-mon[28043]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:04:55.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:55.103 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0[42688]: 2024-04-22T01:04:54.822+0000 7f1744cb1700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T01:04:55.103 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:04:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0[42688]: 2024-04-22T01:04:54.826+0000 7f1740aa4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:04:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:54 smithi203 ceph-mon[31825]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:54 smithi203 ceph-mon[31825]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T01:04:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:54 smithi203 ceph-mon[31825]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T01:04:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:54 smithi203 ceph-mon[31825]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:04:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:55.612 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T01:04:55.612 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T01:04:55.612 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-22T01:04:55.612 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-22T01:04:55.613 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011222 s, 934 MB/s 2024-04-22T01:04:55.613 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-22T01:04:56.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:55 smithi038 ceph-mon[28043]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:04:56.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:55 smithi038 ceph-mon[28043]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T01:04:56.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:55 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:56.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:55 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:55 smithi111 ceph-mon[31057]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:04:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:55 smithi111 ceph-mon[31057]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T01:04:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:55 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:55 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:56.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:55 smithi203 ceph-mon[31825]: from='osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:04:56.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:55 smithi203 ceph-mon[31825]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T01:04:56.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:55 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:56.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:55 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:56.222 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-04-22T01:04:56.462 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:04:57.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] boot 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi038 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:57.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:57.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] boot 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi038 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:04:57.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-22T01:04:57.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:57.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:57.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: osd.0 [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] boot 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi038 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:04:57.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-22T01:04:57.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:57.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:04:57.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 -- 172.21.15.111:0/851949561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c1013f0 msgr2=0x7f7f8c101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 --2- 172.21.15.111:0/851949561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c101870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009990 tx=0x7f7f7c009ca0 comp rx=0 tx=0).stop 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 -- 172.21.15.111:0/851949561 shutdown_connections 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 --2- 172.21.15.111:0/851949561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f8c101db0 0x7f7f8c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 --2- 172.21.15.111:0/851949561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c102820 0x7f7f8c102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 --2- 172.21.15.111:0/851949561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c101870 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:57.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 -- 172.21.15.111:0/851949561 >> 172.21.15.111:0/851949561 conn(0x7f7f8c0fbea0 msgr2=0x7f7f8c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 -- 172.21.15.111:0/851949561 shutdown_connections 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.884+0000 7f7f9301e700 1 -- 172.21.15.111:0/851949561 wait complete. 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 Processor -- start 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 -- start start 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 0x7f7f8c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f8c102820 0x7f7f8c19fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c10ce10 con 0x7f7f8c102820 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c10cc90 con 0x7f7f8c101db0 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f9301e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c10cf90 con 0x7f7f8c1013f0 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 0x7f7f8c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 0x7f7f8c198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60788/0 (socket says 172.21.15.111:60788) 2024-04-22T01:04:57.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.885+0000 7f7f8bfff700 1 -- 172.21.15.111:0/1525686047 learned_addr learned my addr 172.21.15.111:0/1525686047 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f90dba700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f915bb700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f8c102820 0x7f7f8c19fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f8bfff700 1 -- 172.21.15.111:0/1525686047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c1013f0 msgr2=0x7f7f8c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f8bfff700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f8bfff700 1 -- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f8c102820 msgr2=0x7f7f8c19fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f8bfff700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f8c102820 0x7f7f8c19fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f8bfff700 1 -- 172.21.15.111:0/1525686047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c1a00a0 con 0x7f7f8c101db0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f90dba700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f8bfff700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 0x7f7f8c198980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c003890 tx=0x7f7f7c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c027070 con 0x7f7f8c101db0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8c1a0330 con 0x7f7f8c101db0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7c004150 con 0x7f7f8c101db0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00f460 con 0x7f7f8c101db0 2024-04-22T01:04:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.886+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8c1a07e0 con 0x7f7f8c101db0 2024-04-22T01:04:57.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.887+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f8c068f20 con 0x7f7f8c101db0 2024-04-22T01:04:57.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.888+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f7f7c02a070 con 0x7f7f8c101db0 2024-04-22T01:04:57.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.889+0000 7f7f89ffb700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7f780a2b90 0x7f7f780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:04:57.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.889+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f7f7c0ce340 con 0x7f7f8c101db0 2024-04-22T01:04:57.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.889+0000 7f7f90dba700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7f780a2b90 0x7f7f780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:04:57.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.889+0000 7f7f90dba700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7f780a2b90 0x7f7f780a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009dd0 tx=0x7f7f84009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:04:57.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:57.893+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7f7c013070 con 0x7f7f8c101db0 2024-04-22T01:04:58.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:04:58.097+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7f8c0fdf90 con 0x7f7f780a2b90 2024-04-22T01:04:58.098 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:57 smithi111 ceph-mon[31057]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T01:04:58.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:57 smithi038 ceph-mon[28043]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T01:04:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:57 smithi203 ceph-mon[31825]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T01:04:58.953 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:58 smithi111 ceph-mon[31057]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:04:58.953 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:58 smithi111 ceph-mon[31057]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:58.953 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:04:58.953 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:04:58.953 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:04:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:58 smithi038 ceph-mon[28043]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:04:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:58 smithi038 ceph-mon[28043]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:04:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:04:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:04:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:04:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:58 smithi203 ceph-mon[31825]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:04:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:58 smithi203 ceph-mon[31825]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:04:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:04:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:04:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:04:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:00 smithi111 ceph-mon[31057]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:00 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/2607509991' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]: dispatch 2024-04-22T01:05:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:00 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]: dispatch 2024-04-22T01:05:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:00 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]': finished 2024-04-22T01:05:01.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:00 smithi111 ceph-mon[31057]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T01:05:01.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:00 smithi038 ceph-mon[28043]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:00 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/2607509991' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]: dispatch 2024-04-22T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:00 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]: dispatch 2024-04-22T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:00 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]': finished 2024-04-22T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:00 smithi038 ceph-mon[28043]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:00 smithi203 ceph-mon[31825]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:00 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/2607509991' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]: dispatch 2024-04-22T01:05:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:00 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]: dispatch 2024-04-22T01:05:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:00 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01d9f61a-90f2-4b76-98b8-fe7e49315113"}]': finished 2024-04-22T01:05:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:00 smithi203 ceph-mon[31825]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T01:05:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:01 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/1122980369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:02.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:01 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/1122980369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:01 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/1122980369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:03.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:02 smithi111 ceph-mon[31057]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:03.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:02 smithi038 ceph-mon[28043]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:03.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:02 smithi203 ceph-mon[31825]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:04.961 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:04 smithi111 ceph-mon[31057]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:05.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:04 smithi038 ceph-mon[28043]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:05.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:04 smithi203 ceph-mon[31825]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:06 smithi111 ceph-mon[31057]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:07.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:06 smithi038 ceph-mon[28043]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:07.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:06 smithi203 ceph-mon[31825]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T01:05:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:08.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T01:05:08.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:08.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T01:05:08.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:09.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:08 smithi111 ceph-mon[31057]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:09.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:08 smithi111 ceph-mon[31057]: Deploying daemon osd.1 on smithi111 2024-04-22T01:05:09.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:08 smithi038 ceph-mon[28043]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:09.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:08 smithi038 ceph-mon[28043]: Deploying daemon osd.1 on smithi111 2024-04-22T01:05:09.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:08 smithi203 ceph-mon[31825]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:09.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:08 smithi203 ceph-mon[31825]: Deploying daemon osd.1 on smithi111 2024-04-22T01:05:11.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:10 smithi038 ceph-mon[28043]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:11.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:10 smithi203 ceph-mon[31825]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:11.310 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:10 smithi111 ceph-mon[31057]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:12 smithi203 ceph-mon[31825]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:13.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:12 smithi111 ceph-mon[31057]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:13.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:12 smithi038 ceph-mon[28043]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:14.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:14.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:14.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:14.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:14.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:14.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:14.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:14.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:14.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:15.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:15 smithi111 ceph-mon[31057]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:15.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:15 smithi203 ceph-mon[31825]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:15.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:15 smithi038 ceph-mon[28043]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:16.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.164+0000 7f7f89ffb700 1 -- 172.21.15.111:0/1525686047 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7f8c0fdf90 con 0x7f7f780a2b90 2024-04-22T01:05:16.164 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-04-22T01:05:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7f780a2b90 msgr2=0x7f7f780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7f780a2b90 0x7f7f780a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009dd0 tx=0x7f7f84009040 comp rx=0 tx=0).stop 2024-04-22T01:05:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 msgr2=0x7f7f8c198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 0x7f7f8c198980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c003890 tx=0x7f7f7c0047b0 comp rx=0 tx=0).stop 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 shutdown_connections 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7f780a2b90 0x7f7f780a5050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f8c102820 0x7f7f8c19fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c1013f0 0x7f7f8c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 --2- 172.21.15.111:0/1525686047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c101db0 0x7f7f8c198980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 >> 172.21.15.111:0/1525686047 conn(0x7f7f8c0fbea0 msgr2=0x7f7f8c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 shutdown_connections 2024-04-22T01:05:16.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:05:16.167+0000 7f7f9301e700 1 -- 172.21.15.111:0/1525686047 wait complete. 2024-04-22T01:05:17.087 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T01:05:17.090 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-04-22T01:05:17.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T01:05:17.340 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:05:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:17 smithi111 ceph-mon[31057]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:17 smithi111 ceph-mon[31057]: from='osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T01:05:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:17 smithi111 ceph-mon[31057]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:17 smithi038 ceph-mon[28043]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:17 smithi038 ceph-mon[28043]: from='osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:17 smithi038 ceph-mon[28043]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T01:05:17.447 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:17 smithi203 ceph-mon[31825]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:17.447 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:17.447 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:17.447 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:17 smithi203 ceph-mon[31825]: from='osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T01:05:17.447 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:17 smithi203 ceph-mon[31825]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T01:05:18.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:18.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:18.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T01:05:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: from='osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:05:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T01:05:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-mon[31057]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:05:18.396 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1[35617]: 2024-04-22T01:05:18.225+0000 7f218cd69700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T01:05:18.396 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:18 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1[35617]: 2024-04-22T01:05:18.229+0000 7f2188b5c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: from='osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:18.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:18 smithi203 ceph-mon[31825]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: from='osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:18.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:18 smithi038 ceph-mon[28043]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:05:19.026 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T01:05:19.026 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T01:05:19.026 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-22T01:05:19.026 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-22T01:05:19.026 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123395 s, 850 MB/s 2024-04-22T01:05:19.026 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-22T01:05:19.322 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:19 smithi203 ceph-mon[31825]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:19.322 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:19 smithi203 ceph-mon[31825]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:05:19.323 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:19 smithi203 ceph-mon[31825]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T01:05:19.323 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:19 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:19.323 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:19 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:19 smithi111 ceph-mon[31057]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:19 smithi111 ceph-mon[31057]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:05:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:19 smithi111 ceph-mon[31057]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T01:05:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:19 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:19 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:19.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:19 smithi038 ceph-mon[28043]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:19.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:19 smithi038 ceph-mon[28043]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:05:19.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:19 smithi038 ceph-mon[28043]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T01:05:19.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:19.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:19 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:19.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-04-22T01:05:19.905 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:05:20.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:20 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:05:20.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:20 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:05:20.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:20.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:20 smithi203 ceph-mon[31825]: osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] boot 2024-04-22T01:05:20.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:20 smithi203 ceph-mon[31825]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T01:05:20.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:20.369 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:20 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:05:20.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:20 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:05:20.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:20.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:20 smithi111 ceph-mon[31057]: osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] boot 2024-04-22T01:05:20.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:20 smithi111 ceph-mon[31057]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T01:05:20.370 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:20.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:20 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:05:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:20 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:05:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:20 smithi038 ceph-mon[28043]: osd.1 [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] boot 2024-04-22T01:05:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:20 smithi038 ceph-mon[28043]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T01:05:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 -- 172.21.15.203:0/1772275336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103e30 msgr2=0x7f9e2010a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/1772275336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103e30 0x7f9e2010a6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800b0a0 tx=0x7f9e1800b3b0 comp rx=0 tx=0).stop 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 -- 172.21.15.203:0/1772275336 shutdown_connections 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/1772275336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103e30 0x7f9e2010a6e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/1772275336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20102290 0x7f9e201026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/1772275336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e20103490 0x7f9e201038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 -- 172.21.15.203:0/1772275336 >> 172.21.15.203:0/1772275336 conn(0x7f9e200fd830 msgr2=0x7f9e200ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:05:21.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 -- 172.21.15.203:0/1772275336 shutdown_connections 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 -- 172.21.15.203:0/1772275336 wait complete. 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.389+0000 7f9e26c5e700 1 Processor -- start 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 -- start start 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e20102290 0x7f9e201984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 0x7f9e201989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20103e30 0x7f9e2019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010df10 con 0x7f9e20103e30 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010dd90 con 0x7f9e20102290 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010e090 con 0x7f9e20103490 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 0x7f9e201989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:21.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 0x7f9e201989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:55304/0 (socket says 172.21.15.203:55304) 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 -- 172.21.15.203:0/3742975610 learned_addr learned my addr 172.21.15.203:0/3742975610 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e249fa700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e20102290 0x7f9e201984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 -- 172.21.15.203:0/3742975610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e20102290 msgr2=0x7f9e201984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e20102290 0x7f9e201984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 -- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20103e30 msgr2=0x7f9e2019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e251fb700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20103e30 0x7f9e2019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20103e30 0x7f9e2019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 -- 172.21.15.203:0/3742975610 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e201a0100 con 0x7f9e20103490 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e251fb700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20103e30 0x7f9e2019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e17fff700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 0x7f9e201989e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000d9b0 tx=0x7f9e1000dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10017070 con 0x7f9e20103490 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e10004d10 con 0x7f9e20103490 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e201a03f0 con 0x7f9e20103490 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.390+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10005500 con 0x7f9e20103490 2024-04-22T01:05:21.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.391+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e201a09b0 con 0x7f9e20103490 2024-04-22T01:05:21.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.392+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e20068f20 con 0x7f9e20103490 2024-04-22T01:05:21.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.393+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f9e10004750 con 0x7f9e20103490 2024-04-22T01:05:21.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.393+0000 7f9e15ffb700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9e080a2c10 0x7f9e080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:21.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.393+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f9e100cd6c0 con 0x7f9e20103490 2024-04-22T01:05:21.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.394+0000 7f9e249fa700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9e080a2c10 0x7f9e080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:21.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.394+0000 7f9e249fa700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9e080a2c10 0x7f9e080a50d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c006fd0 tx=0x7f9e1c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:05:21.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.398+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9e10020020 con 0x7f9e20103490 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi111 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:21.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:21.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:21.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:21.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:21.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:21.604+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9e20062670 con 0x7f9e080a2c10 2024-04-22T01:05:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi111 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:21.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:21.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:21.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi111 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:21.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:21.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:21.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:22.497 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:22 smithi203 ceph-mon[31825]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:05:22.498 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:05:22.498 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:05:22.498 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:22 smithi111 ceph-mon[31057]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:05:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:05:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:05:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:22.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:22 smithi038 ceph-mon[28043]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:05:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:05:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:05:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:23.305 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:23 smithi203 ceph-mon[31825]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:23 smithi111 ceph-mon[31057]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:23.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:23 smithi038 ceph-mon[28043]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:24.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:24 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/1185107681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]: dispatch 2024-04-22T01:05:24.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:24 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]: dispatch 2024-04-22T01:05:24.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:24 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]': finished 2024-04-22T01:05:24.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:24 smithi203 ceph-mon[31825]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T01:05:24.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:24 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/1185107681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]: dispatch 2024-04-22T01:05:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:24 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]: dispatch 2024-04-22T01:05:24.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:24 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]': finished 2024-04-22T01:05:24.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:24 smithi111 ceph-mon[31057]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T01:05:24.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:24 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/1185107681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]: dispatch 2024-04-22T01:05:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:24 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]: dispatch 2024-04-22T01:05:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:24 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ce3eb04-5c85-4391-92f6-dd636ae03c6e"}]': finished 2024-04-22T01:05:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:24 smithi038 ceph-mon[28043]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T01:05:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:25.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:25 smithi203 ceph-mon[31825]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:25.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:25 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/2530245764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:25 smithi111 ceph-mon[31057]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:25 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/2530245764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:25.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:25 smithi038 ceph-mon[28043]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:25.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:25 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/2530245764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:27.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:27 smithi203 ceph-mon[31825]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:27.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:27 smithi111 ceph-mon[31057]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:27.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:27 smithi038 ceph-mon[28043]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:29.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:29 smithi203 ceph-mon[31825]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:29.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:29 smithi111 ceph-mon[31057]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:29.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:29 smithi038 ceph-mon[28043]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:31 smithi203 ceph-mon[31825]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T01:05:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:31 smithi111 ceph-mon[31057]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T01:05:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:31 smithi038 ceph-mon[28043]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:32 smithi203 ceph-mon[31825]: Deploying daemon osd.2 on smithi203 2024-04-22T01:05:32.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:32 smithi111 ceph-mon[31057]: Deploying daemon osd.2 on smithi203 2024-04-22T01:05:32.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:32 smithi038 ceph-mon[28043]: Deploying daemon osd.2 on smithi203 2024-04-22T01:05:33.395 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:33 smithi203 ceph-mon[31825]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:33.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:33 smithi111 ceph-mon[31057]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:33.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:33 smithi038 ceph-mon[28043]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:35.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:35 smithi111 ceph-mon[31057]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:35.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:35 smithi038 ceph-mon[28043]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:35.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:35 smithi203 ceph-mon[31825]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:37.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:37 smithi203 ceph-mon[31825]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:37.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:37 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:37.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:37 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:37.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:37 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:37 smithi111 ceph-mon[31057]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:37 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:37 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:37 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:37 smithi038 ceph-mon[28043]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:37 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:37 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:37 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:39 smithi203 ceph-mon[31825]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:39 smithi111 ceph-mon[31057]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:39.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:39 smithi038 ceph-mon[28043]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:39.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.974+0000 7f9e15ffb700 1 -- 172.21.15.203:0/3742975610 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9e20062670 con 0x7f9e080a2c10 2024-04-22T01:05:39.975 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-04-22T01:05:39.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9e080a2c10 msgr2=0x7f9e080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:39.977 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9e080a2c10 0x7f9e080a50d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c006fd0 tx=0x7f9e1c008040 comp rx=0 tx=0).stop 2024-04-22T01:05:39.977 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 msgr2=0x7f9e201989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:39.977 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 0x7f9e201989e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000d9b0 tx=0x7f9e1000dd70 comp rx=0 tx=0).stop 2024-04-22T01:05:39.977 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 shutdown_connections 2024-04-22T01:05:39.977 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9e080a2c10 0x7f9e080a50d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:39.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9e20103e30 0x7f9e2019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:39.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e20102290 0x7f9e201984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:39.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 --2- 172.21.15.203:0/3742975610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e20103490 0x7f9e201989e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:39.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 >> 172.21.15.203:0/3742975610 conn(0x7f9e200fd830 msgr2=0x7f9e20106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:05:39.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 shutdown_connections 2024-04-22T01:05:39.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:05:39.976+0000 7f9e26c5e700 1 -- 172.21.15.203:0/3742975610 wait complete. 2024-04-22T01:05:40.597 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T01:05:40.599 INFO:tasks.cephadm:Deploying osd.3 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-22T01:05:40.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T01:05:40.859 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:05:41.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:41.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: from='osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T01:05:41.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T01:05:41.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: from='osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T01:05:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T01:05:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: from='osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:41.263 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:42.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:41 smithi038 ceph-mon[28043]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T01:05:42.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:41 smithi038 ceph-mon[28043]: from='osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:05:42.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:41 smithi038 ceph-mon[28043]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T01:05:42.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:41 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:42.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:41 smithi038 ceph-mon[28043]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:05:42.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:41 smithi203 ceph-mon[31825]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T01:05:42.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:41 smithi203 ceph-mon[31825]: from='osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:05:42.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:41 smithi203 ceph-mon[31825]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T01:05:42.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:41 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:42.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:41 smithi203 ceph-mon[31825]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:05:42.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:41 smithi111 ceph-mon[31057]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T01:05:42.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:41 smithi111 ceph-mon[31057]: from='osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:05:42.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:41 smithi111 ceph-mon[31057]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T01:05:42.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:41 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:42.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:41 smithi111 ceph-mon[31057]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:05:42.607 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2[36960]: 2024-04-22T01:05:42.381+0000 7f63d219e700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T01:05:42.607 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2[36960]: 2024-04-22T01:05:42.385+0000 7f63c9788700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:05:42.933 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T01:05:42.933 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T01:05:42.933 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T01:05:42.934 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T01:05:42.934 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155752 s, 673 MB/s 2024-04-22T01:05:42.934 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T01:05:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-mon[31825]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-mon[31825]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:05:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-mon[31825]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T01:05:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:42 smithi203 ceph-mon[31825]: from='osd.2 ' entity='osd.2' 2024-04-22T01:05:43.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:42 smithi038 ceph-mon[28043]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:43.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:42 smithi038 ceph-mon[28043]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:05:43.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:42 smithi038 ceph-mon[28043]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T01:05:43.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:42 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:43.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:42 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:43.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:42 smithi038 ceph-mon[28043]: from='osd.2 ' entity='osd.2' 2024-04-22T01:05:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:42 smithi111 ceph-mon[31057]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:42 smithi111 ceph-mon[31057]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:05:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:42 smithi111 ceph-mon[31057]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T01:05:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:42 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:42 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:42 smithi111 ceph-mon[31057]: from='osd.2 ' entity='osd.2' 2024-04-22T01:05:43.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-22T01:05:43.890 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:05:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:44 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:05:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:44 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:05:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:44 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:44 smithi203 ceph-mon[31825]: osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] boot 2024-04-22T01:05:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:44 smithi203 ceph-mon[31825]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T01:05:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:44 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:44 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:05:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:44 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:05:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:44 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:44 smithi111 ceph-mon[31057]: osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] boot 2024-04-22T01:05:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:44 smithi111 ceph-mon[31057]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T01:05:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:44 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:44.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:44 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:05:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:44 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:05:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:44 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:44 smithi038 ceph-mon[28043]: osd.2 [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] boot 2024-04-22T01:05:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:44 smithi038 ceph-mon[28043]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T01:05:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:44 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi203 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:45.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:45.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi203 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:45.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 -- 172.21.15.38:0/2679691476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e50103e60 msgr2=0x7f5e5010a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 --2- 172.21.15.38:0/2679691476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e50103e60 0x7f5e5010a710 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800b0d0 tx=0x7f5e4800b3e0 comp rx=0 tx=0).stop 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 -- 172.21.15.38:0/2679691476 shutdown_connections 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 --2- 172.21.15.38:0/2679691476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e50103e60 0x7f5e5010a710 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 --2- 172.21.15.38:0/2679691476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e501022c0 0x7f5e501026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 --2- 172.21.15.38:0/2679691476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e501034c0 0x7f5e50103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 -- 172.21.15.38:0/2679691476 >> 172.21.15.38:0/2679691476 conn(0x7f5e500fd880 msgr2=0x7f5e500ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 -- 172.21.15.38:0/2679691476 shutdown_connections 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.558+0000 7f5e566b1700 1 -- 172.21.15.38:0/2679691476 wait complete. 2024-04-22T01:05:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 Processor -- start 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 -- start start 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e501022c0 0x7f5e501983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 0x7f5e50198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50103e60 0x7f5e5019f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5010de90 con 0x7f5e501034c0 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5010dd10 con 0x7f5e50103e60 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e566b1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5010e010 con 0x7f5e501022c0 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e4f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 0x7f5e50198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e4f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 0x7f5e50198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56926/0 (socket says 172.21.15.38:56926) 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e4f7fe700 1 -- 172.21.15.38:0/1181568969 learned_addr learned my addr 172.21.15.38:0/1181568969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e54c4e700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50103e60 0x7f5e5019f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.559+0000 7f5e4ffff700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e501022c0 0x7f5e501983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e54c4e700 1 -- 172.21.15.38:0/1181568969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e501022c0 msgr2=0x7f5e501983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e54c4e700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e501022c0 0x7f5e501983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e54c4e700 1 -- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 msgr2=0x7f5e50198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e54c4e700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 0x7f5e50198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e54c4e700 1 -- 172.21.15.38:0/1181568969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e501a0020 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e4f7fe700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 0x7f5e50198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e54c4e700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50103e60 0x7f5e5019f980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800d010 tx=0x7f5e48007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48021070 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e501a02b0 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4801db30 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4801bba0 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.560+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e501a0760 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.561+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e50068f20 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.562+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f5e48024070 con 0x7f5e50103e60 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.563+0000 7f5e4d7fa700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5e380a2b90 0x7f5e380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:05:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.563+0000 7f5e4ffff700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5e380a2b90 0x7f5e380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:05:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.563+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f5e4800f020 con 0x7f5e50103e60 2024-04-22T01:05:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.564+0000 7f5e4ffff700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5e380a2b90 0x7f5e380a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e50103320 tx=0x7f5e4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:05:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.567+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5e4802b020 con 0x7f5e50103e60 2024-04-22T01:05:45.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T01:05:45.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi203 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:05:45.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:05:45.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:45.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:05:45.647 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:45.783+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5e50062730 con 0x7f5e380a2b90 2024-04-22T01:05:46.405 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:45 smithi038 sudo[47141]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T01:05:46.405 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:45 smithi038 sudo[47141]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:46.405 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:45 smithi038 sudo[47141]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:46.405 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:45 smithi038 sudo[47141]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:46.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:05:46.418+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5e48099b30 con 0x7f5e50103e60 2024-04-22T01:05:46.675 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:46 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T01:05:46.675 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:46 smithi111 ceph-mon[31057]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T01:05:46.675 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:46 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:05:46.675 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:46 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:05:46.676 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:46 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:46 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T01:05:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:46 smithi203 ceph-mon[31825]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T01:05:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:46 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:05:46.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:46 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:05:46.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:46 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:46.770 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:46 smithi038 sudo[47199]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T01:05:46.770 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:46 smithi038 sudo[47199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:46.771 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:46 smithi038 sudo[47199]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:46.771 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:05:46 smithi038 sudo[47199]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:46.771 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T01:05:46.771 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:46 smithi038 ceph-mon[28043]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T01:05:46.772 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:05:46.772 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:05:46.772 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:46.926 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39301]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T01:05:46.926 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39301]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:46.926 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39301]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:46.927 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39301]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:46.927 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T01:05:46.927 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:46.927 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39304]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:46.927 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:05:46 smithi111 sudo[39304]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:47.447 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40854]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T01:05:47.447 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:47.447 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40854]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:47.447 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40854]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:47.448 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:47 smithi203 ceph-mon[31825]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:47.448 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:47 smithi203 ceph-mon[31825]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:05:47.448 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:47 smithi203 ceph-mon[31825]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T01:05:47.448 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:47 smithi203 ceph-mon[31825]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T01:05:47.448 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:47.448 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 ceph-mon[28043]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 ceph-mon[28043]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 ceph-mon[28043]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 ceph-mon[28043]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:47.766 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40857]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T01:05:47.766 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40857]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:47.766 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40857]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:47.766 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:05:47 smithi203 sudo[40857]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:47 smithi111 ceph-mon[31057]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:47 smithi111 ceph-mon[31057]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:05:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:47 smithi111 ceph-mon[31057]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T01:05:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:47 smithi111 ceph-mon[31057]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T01:05:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:47.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 sudo[47251]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T01:05:47.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 sudo[47251]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:47.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 sudo[47251]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:47.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:47 smithi038 sudo[47251]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:48.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:48 smithi038 sudo[47288]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T01:05:48.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:48 smithi038 sudo[47288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:48.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:48 smithi038 sudo[47288]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:48.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:48 smithi038 sudo[47288]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:48.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40860]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T01:05:48.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:48.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40860]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:48.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40860]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40863]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T01:05:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40863]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40863]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:48 smithi203 sudo[40863]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:49.367 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39307]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T01:05:49.367 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39307]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:49.367 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39307]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:49.368 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39307]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:49.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39310]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T01:05:49.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39310]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T01:05:49.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39310]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T01:05:49.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 sudo[39310]: pam_unix(sudo:session): session closed for user root 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:49.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:49.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:49.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:49.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:49.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:49.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:49.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:49.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:49.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:49.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/406646224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e29b0ec6-0914-46b2-9887-c808bd3ab6b2"}]: dispatch 2024-04-22T01:05:50.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/406646224' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e29b0ec6-0914-46b2-9887-c808bd3ab6b2"}]': finished 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:05:50.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:50 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/567102197' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/406646224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e29b0ec6-0914-46b2-9887-c808bd3ab6b2"}]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/406646224' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e29b0ec6-0914-46b2-9887-c808bd3ab6b2"}]': finished 2024-04-22T01:05:50.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T01:05:50.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:05:50.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:50 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/567102197' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/406646224' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e29b0ec6-0914-46b2-9887-c808bd3ab6b2"}]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T01:05:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/406646224' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e29b0ec6-0914-46b2-9887-c808bd3ab6b2"}]': finished 2024-04-22T01:05:50.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T01:05:50.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:05:50.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:50 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/567102197' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:05:51.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:51 smithi111 ceph-mon[31057]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:51.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:51 smithi038 ceph-mon[28043]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:51.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:51 smithi203 ceph-mon[31825]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:53.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:53 smithi111 ceph-mon[31057]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:53 smithi038 ceph-mon[28043]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:53.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:53 smithi203 ceph-mon[31825]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:55.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:55 smithi111 ceph-mon[31057]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:55.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:55 smithi038 ceph-mon[28043]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:55.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:55 smithi203 ceph-mon[31825]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:57 smithi111 ceph-mon[31057]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T01:05:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:57.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:57 smithi038 ceph-mon[28043]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T01:05:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:57 smithi203 ceph-mon[31825]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T01:05:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:05:58.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:58 smithi038 ceph-mon[28043]: Deploying daemon osd.3 on smithi038 2024-04-22T01:05:58.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:58 smithi203 ceph-mon[31825]: Deploying daemon osd.3 on smithi038 2024-04-22T01:05:58.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:58 smithi111 ceph-mon[31057]: Deploying daemon osd.3 on smithi038 2024-04-22T01:05:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:05:59 smithi038 ceph-mon[28043]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:59.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:05:59 smithi203 ceph-mon[31825]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:05:59.796 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:05:59 smithi111 ceph-mon[31057]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:01.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:01 smithi038 ceph-mon[28043]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:01.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:01 smithi203 ceph-mon[31825]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:01.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:01 smithi111 ceph-mon[31057]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:03 smithi203 ceph-mon[31825]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:03.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:03 smithi038 ceph-mon[28043]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:03.839 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:03.839 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:03.839 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:03 smithi111 ceph-mon[31057]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:05.716 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:05 smithi038 ceph-mon[28043]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:05.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:05 smithi203 ceph-mon[31825]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:05 smithi111 ceph-mon[31057]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:06.742 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 3 on host 'smithi038' 2024-04-22T01:06:06.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.741+0000 7f5e4d7fa700 1 -- 172.21.15.38:0/1181568969 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5e50062730 con 0x7f5e380a2b90 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5e380a2b90 msgr2=0x7f5e380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5e380a2b90 0x7f5e380a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e50103320 tx=0x7f5e4000b410 comp rx=0 tx=0).stop 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50103e60 msgr2=0x7f5e5019f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50103e60 0x7f5e5019f980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800d010 tx=0x7f5e48007a20 comp rx=0 tx=0).stop 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 shutdown_connections 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5e380a2b90 0x7f5e380a5050 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50103e60 0x7f5e5019f980 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e501022c0 0x7f5e501983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 --2- 172.21.15.38:0/1181568969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e501034c0 0x7f5e50198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.743+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 >> 172.21.15.38:0/1181568969 conn(0x7f5e500fd880 msgr2=0x7f5e50106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:06:06.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.744+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 shutdown_connections 2024-04-22T01:06:06.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:06:06.744+0000 7f5e566b1700 1 -- 172.21.15.38:0/1181568969 wait complete. 2024-04-22T01:06:07.395 DEBUG:teuthology.orchestra.run.smithi038:osd.3> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T01:06:07.398 INFO:tasks.cephadm:Deploying osd.4 on smithi111 with /dev/vg_nvme/lv_3... 2024-04-22T01:06:07.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T01:06:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:06:07.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:07 smithi038 ceph-mon[28043]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:07 smithi038 ceph-mon[28043]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T01:06:07.764 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:07 smithi111 ceph-mon[31057]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:07.764 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:07.764 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:07.764 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:07 smithi111 ceph-mon[31057]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T01:06:07.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:07 smithi203 ceph-mon[31825]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:07.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:07.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:07.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:07 smithi203 ceph-mon[31825]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:08 smithi038 ceph-mon[28043]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:08 smithi038 ceph-mon[28043]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:08 smithi038 ceph-mon[28043]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:08 smithi203 ceph-mon[31825]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T01:06:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:08 smithi203 ceph-mon[31825]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T01:06:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:08 smithi203 ceph-mon[31825]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:06:08.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:08.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:08.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:08.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:08 smithi111 ceph-mon[31057]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T01:06:08.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:08 smithi111 ceph-mon[31057]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T01:06:08.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:08 smithi111 ceph-mon[31057]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:06:08.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.405 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:06:09.139+0000 7fbe030ef700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T01:06:09.405 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:06:09.147+0000 7fbdfaeda700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:06:09.713 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-mon[28043]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:09.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-mon[28043]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:06:09.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-mon[28043]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T01:06:09.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:09 smithi038 ceph-mon[28043]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' 2024-04-22T01:06:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:09 smithi203 ceph-mon[31825]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:09 smithi203 ceph-mon[31825]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:06:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:09 smithi203 ceph-mon[31825]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T01:06:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:09 smithi203 ceph-mon[31825]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' 2024-04-22T01:06:09.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:09 smithi111 ceph-mon[31057]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:09.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:09 smithi111 ceph-mon[31057]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:06:09.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:09 smithi111 ceph-mon[31057]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T01:06:09.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:09.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:09 smithi111 ceph-mon[31057]: from='osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332]' entity='osd.3' 2024-04-22T01:06:09.920 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T01:06:09.920 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T01:06:09.920 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-22T01:06:09.920 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-22T01:06:09.921 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130965 s, 801 MB/s 2024-04-22T01:06:09.921 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-22T01:06:10.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-04-22T01:06:10.720 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:06:10.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:06:10.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:10.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] boot 2024-04-22T01:06:10.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T01:06:10.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:10.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:10 smithi038 ceph-mon[28043]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] boot 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:10.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:10 smithi203 ceph-mon[31825]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T01:06:10.833 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:06:10.868 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:06:10.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:06:10.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:10.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: osd.3 [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] boot 2024-04-22T01:06:10.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T01:06:10.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T01:06:10.869 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:10 smithi111 ceph-mon[31057]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:11.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:11.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:11.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:11 smithi203 ceph-mon[31825]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:11.812 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:11.813 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.813 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:11 smithi111 ceph-mon[31057]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:11.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:11.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:11.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:11.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:11.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:11 smithi038 ceph-mon[28043]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T01:06:12.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 -- 172.21.15.111:0/2804601185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 msgr2=0x7f03f8107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 --2- 172.21.15.111:0/2804601185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f8107a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009990 tx=0x7f03ec009ca0 comp rx=0 tx=0).stop 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 -- 172.21.15.111:0/2804601185 shutdown_connections 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 --2- 172.21.15.111:0/2804601185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 0x7f03f810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 --2- 172.21.15.111:0/2804601185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03f8102d60 0x7f03f8105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 --2- 172.21.15.111:0/2804601185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f8107a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 -- 172.21.15.111:0/2804601185 >> 172.21.15.111:0/2804601185 conn(0x7f03f80fc790 msgr2=0x7f03f80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 -- 172.21.15.111:0/2804601185 shutdown_connections 2024-04-22T01:06:12.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.561+0000 7f03fccfc700 1 -- 172.21.15.111:0/2804601185 wait complete. 2024-04-22T01:06:12.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 Processor -- start 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 -- start start 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03f8102d60 0x7f03f8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 0x7f03f819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f03f810dee0 con 0x7f03f8102d60 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03f810dd60 con 0x7f03f8105690 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03fccfc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03f810e060 con 0x7f03f8107fc0 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f81989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43328/0 (socket says 172.21.15.111:43328) 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 -- 172.21.15.111:0/3253236988 learned_addr learned my addr 172.21.15.111:0/3253236988 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f659c700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03f8102d60 0x7f03f8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:12.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f6d9d700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 0x7f03f819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 -- 172.21.15.111:0/3253236988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 msgr2=0x7f03f819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 0x7f03f819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 -- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03f8102d60 msgr2=0x7f03f8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03f8102d60 0x7f03f8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f5d9b700 1 -- 172.21.15.111:0/3253236988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f81a00c0 con 0x7f03f8105690 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.562+0000 7f03f6d9d700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 0x7f03f819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.563+0000 7f03f5d9b700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f81989a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f03ec005910 tx=0x7f03ec009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.563+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec027070 con 0x7f03f8105690 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.563+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f81a0350 con 0x7f03f8105690 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.563+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03ec004150 con 0x7f03f8105690 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.563+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00f460 con 0x7f03f8105690 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.563+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f81a07a0 con 0x7f03f8105690 2024-04-22T01:06:12.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.564+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f8068f20 con 0x7f03f8105690 2024-04-22T01:06:12.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.565+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f03ec02a070 con 0x7f03f8105690 2024-04-22T01:06:12.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.565+0000 7f03e77fe700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f03e00a2c20 0x7f03e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:12.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.566+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f03ec0cf440 con 0x7f03f8105690 2024-04-22T01:06:12.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.566+0000 7f03f659c700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f03e00a2c20 0x7f03e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:12.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.566+0000 7f03f659c700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f03e00a2c20 0x7f03e00a50e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f03e8005950 tx=0x7f03e800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:06:12.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.570+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f03ec031070 con 0x7f03f8105690 2024-04-22T01:06:12.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:12.770+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f03f8062670 con 0x7f03e00a2c20 2024-04-22T01:06:12.771 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:12 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi038 2024-04-22T01:06:12.771 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:12 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-22T01:06:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:12 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi038 2024-04-22T01:06:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:12 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-22T01:06:13.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:12 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi038 2024-04-22T01:06:13.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:12 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-22T01:06:13.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:13 smithi111 ceph-mon[31057]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:13.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:13 smithi111 ceph-mon[31057]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T01:06:13.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:06:13.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:06:13.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:13 smithi038 ceph-mon[28043]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:13 smithi038 ceph-mon[28043]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T01:06:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:06:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:06:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:13 smithi203 ceph-mon[31825]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:13 smithi203 ceph-mon[31825]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T01:06:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:06:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:06:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:14.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:14 smithi111 ceph-mon[31057]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:06:14.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:14 smithi111 ceph-mon[31057]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T01:06:14.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:14 smithi038 ceph-mon[28043]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:06:14.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:14 smithi038 ceph-mon[28043]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T01:06:15.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:14 smithi203 ceph-mon[31825]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:06:15.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:14 smithi203 ceph-mon[31825]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T01:06:15.882 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:15 smithi111 ceph-mon[31057]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:15.882 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:15 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/536291957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]: dispatch 2024-04-22T01:06:15.882 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:15 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]: dispatch 2024-04-22T01:06:15.882 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:15 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]': finished 2024-04-22T01:06:15.883 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:15 smithi111 ceph-mon[31057]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T01:06:15.883 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:15 smithi038 ceph-mon[28043]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:15 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/536291957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]: dispatch 2024-04-22T01:06:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:15 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]: dispatch 2024-04-22T01:06:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:15 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]': finished 2024-04-22T01:06:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:15 smithi038 ceph-mon[28043]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T01:06:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:15 smithi203 ceph-mon[31825]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:15 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/536291957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]: dispatch 2024-04-22T01:06:16.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:15 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]: dispatch 2024-04-22T01:06:16.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:15 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05e84962-4508-4457-a9ea-c52fd31377b4"}]': finished 2024-04-22T01:06:16.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:15 smithi203 ceph-mon[31825]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T01:06:16.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:16 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/681591044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:06:16.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:16 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/681591044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:06:17.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:16 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/681591044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:06:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:17 smithi111 ceph-mon[31057]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-22T01:06:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:17 smithi038 ceph-mon[28043]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-22T01:06:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:17 smithi203 ceph-mon[31825]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-22T01:06:19.822 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:19 smithi111 ceph-mon[31057]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:06:19.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:19 smithi038 ceph-mon[28043]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:06:20.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:19 smithi203 ceph-mon[31825]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:06:21.815 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:21 smithi111 ceph-mon[31057]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-22T01:06:21.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:21 smithi038 ceph-mon[28043]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-22T01:06:22.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:21 smithi203 ceph-mon[31825]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-22T01:06:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:23 smithi111 ceph-mon[31057]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T01:06:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T01:06:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:23 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:23 smithi038 ceph-mon[28043]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T01:06:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T01:06:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:23 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:24.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:23 smithi203 ceph-mon[31825]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T01:06:24.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T01:06:24.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:23 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:24.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:24 smithi111 ceph-mon[31057]: Deploying daemon osd.4 on smithi111 2024-04-22T01:06:24.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:24 smithi038 ceph-mon[28043]: Deploying daemon osd.4 on smithi111 2024-04-22T01:06:25.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:24 smithi203 ceph-mon[31825]: Deploying daemon osd.4 on smithi111 2024-04-22T01:06:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:25 smithi111 ceph-mon[31057]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:06:25.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:25 smithi038 ceph-mon[28043]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:06:26.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:25 smithi203 ceph-mon[31825]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:06:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:27 smithi111 ceph-mon[31057]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:27.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:27 smithi038 ceph-mon[28043]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:28.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:27 smithi203 ceph-mon[31825]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:29 smithi111 ceph-mon[31057]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:29.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:29 smithi038 ceph-mon[28043]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:29.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:29.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:29.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:29 smithi203 ceph-mon[31825]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:31 smithi111 ceph-mon[31057]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:31 smithi038 ceph-mon[28043]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:31 smithi203 ceph-mon[31825]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:32.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.392+0000 7f03e77fe700 1 -- 172.21.15.111:0/3253236988 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f03f8062670 con 0x7f03e00a2c20 2024-04-22T01:06:32.392 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 4 on host 'smithi111' 2024-04-22T01:06:32.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f03e00a2c20 msgr2=0x7f03e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:32.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f03e00a2c20 0x7f03e00a50e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f03e8005950 tx=0x7f03e800a300 comp rx=0 tx=0).stop 2024-04-22T01:06:32.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 msgr2=0x7f03f81989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f81989a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f03ec005910 tx=0x7f03ec009580 comp rx=0 tx=0).stop 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 shutdown_connections 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f03e00a2c20 0x7f03e00a50e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f8107fc0 0x7f03f819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03f8102d60 0x7f03f8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 --2- 172.21.15.111:0/3253236988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8105690 0x7f03f81989a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 >> 172.21.15.111:0/3253236988 conn(0x7f03f80fc790 msgr2=0x7f03f81036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 shutdown_connections 2024-04-22T01:06:32.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:06:32.395+0000 7f03fccfc700 1 -- 172.21.15.111:0/3253236988 wait complete. 2024-04-22T01:06:33.092 DEBUG:teuthology.orchestra.run.smithi111:osd.4> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T01:06:33.094 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-04-22T01:06:33.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T01:06:33.340 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:06:33.610 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:33 smithi111 ceph-mon[31057]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:33.610 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:33.610 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:33.610 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:33.610 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:33 smithi111 ceph-mon[31057]: from='osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T01:06:33.610 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:33 smithi111 ceph-mon[31057]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T01:06:33.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:33 smithi038 ceph-mon[28043]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:33.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:33.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:33.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:33.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:33 smithi038 ceph-mon[28043]: from='osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T01:06:33.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:33 smithi038 ceph-mon[28043]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T01:06:33.693 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:33 smithi203 ceph-mon[31825]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:33.693 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:33.693 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:33.694 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:33.694 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:33 smithi203 ceph-mon[31825]: from='osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T01:06:33.694 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:33 smithi203 ceph-mon[31825]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T01:06:34.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-mon[31057]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T01:06:34.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-mon[31057]: from='osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:06:34.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-mon[31057]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T01:06:34.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:34.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-mon[31057]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:06:34.646 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:06:34.393+0000 7f2ba5713700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T01:06:34.647 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:06:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:06:34.400+0000 7f2ba1506700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:06:34.647 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:34 smithi203 ceph-mon[31825]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T01:06:34.647 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:34 smithi203 ceph-mon[31825]: from='osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:06:34.648 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:34 smithi203 ceph-mon[31825]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T01:06:34.648 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:34 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:34.648 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:34 smithi203 ceph-mon[31825]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:06:34.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:34 smithi038 ceph-mon[28043]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T01:06:34.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:34 smithi038 ceph-mon[28043]: from='osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:06:34.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:34 smithi038 ceph-mon[28043]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T01:06:34.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:34 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:34.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:34 smithi038 ceph-mon[28043]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:06:35.305 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T01:06:35.306 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T01:06:35.306 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-22T01:06:35.306 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-22T01:06:35.306 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120528 s, 870 MB/s 2024-04-22T01:06:35.306 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-22T01:06:35.600 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:35 smithi203 ceph-mon[31825]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:35.600 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:35 smithi203 ceph-mon[31825]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:06:35.600 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:35 smithi203 ceph-mon[31825]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T01:06:35.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:35 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:35 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:35 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.862 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:35 smithi111 ceph-mon[31057]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:35.862 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:35 smithi111 ceph-mon[31057]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:06:35.862 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:35 smithi111 ceph-mon[31057]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T01:06:35.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:35 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:35 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:35 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:35 smithi038 ceph-mon[28043]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:35.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:35 smithi038 ceph-mon[28043]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:06:35.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:35 smithi038 ceph-mon[28043]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T01:06:35.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:35 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:35 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:35 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:35.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-04-22T01:06:36.224 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:06:36.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] boot 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] boot 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:36.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:36.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:06:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:06:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: osd.4 [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] boot 2024-04-22T01:06:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:06:36.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:37.577 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:37 smithi203 ceph-mon[31825]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:37.577 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:37 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi111 2024-04-22T01:06:37.577 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:37 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-22T01:06:37.577 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:37 smithi203 ceph-mon[31825]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 -- 172.21.15.203:0/1522762675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec1023b0 msgr2=0x7f91ec102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 --2- 172.21.15.203:0/1522762675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec1023b0 0x7f91ec102810 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009990 tx=0x7f91dc009ca0 comp rx=0 tx=0).stop 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 -- 172.21.15.203:0/1522762675 shutdown_connections 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 --2- 172.21.15.203:0/1522762675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec102d50 0x7f91ec109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 --2- 172.21.15.203:0/1522762675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ec1011b0 0x7f91ec1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 --2- 172.21.15.203:0/1522762675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec1023b0 0x7f91ec102810 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 -- 172.21.15.203:0/1522762675 >> 172.21.15.203:0/1522762675 conn(0x7f91ec0fc790 msgr2=0x7f91ec0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 -- 172.21.15.203:0/1522762675 shutdown_connections 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.816+0000 7f91f40ae700 1 -- 172.21.15.203:0/1522762675 wait complete. 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 Processor -- start 2024-04-22T01:06:37.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 -- start start 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec1011b0 0x7f91ec06b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ec1023b0 0x7f91ec19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 0x7f91ec1a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec10cd80 con 0x7f91ec1011b0 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec10cc00 con 0x7f91ec1023b0 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f40ae700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec10cf00 con 0x7f91ec102d50 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 0x7f91ec1a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 0x7f91ec1a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:41364/0 (socket says 172.21.15.203:41364) 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 -- 172.21.15.203:0/340880874 learned_addr learned my addr 172.21.15.203:0/340880874 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f1649700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ec1023b0 0x7f91ec19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:37.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f1e4a700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec1011b0 0x7f91ec06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 -- 172.21.15.203:0/340880874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ec1023b0 msgr2=0x7f91ec19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ec1023b0 0x7f91ec19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 -- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec1011b0 msgr2=0x7f91ec06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec1011b0 0x7f91ec06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 -- 172.21.15.203:0/340880874 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec1a2350 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f1e4a700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec1011b0 0x7f91ec06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.817+0000 7f91f264b700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 0x7f91ec1a1b70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91e400ada0 tx=0x7f91e400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.818+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4017020 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.818+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1a2640 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.818+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91e400a0d0 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.818+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4005480 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.818+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec1a2b00 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.819+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec068f20 con 0x7f91ec102d50 2024-04-22T01:06:37.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.820+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f91e4005620 con 0x7f91ec102d50 2024-04-22T01:06:37.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.820+0000 7f91e2ffd700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d80a2c20 0x7f91d80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:06:37.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.820+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f91e40cd760 con 0x7f91ec102d50 2024-04-22T01:06:37.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.820+0000 7f91f1e4a700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d80a2c20 0x7f91d80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:06:37.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.821+0000 7f91f1e4a700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d80a2c20 0x7f91d80a50e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f91e8005fd0 tx=0x7f91e8005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:06:37.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:37.825+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f91e4021020 con 0x7f91ec102d50 2024-04-22T01:06:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:37 smithi111 ceph-mon[31057]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:37 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi111 2024-04-22T01:06:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:37 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-22T01:06:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:37 smithi111 ceph-mon[31057]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T01:06:37.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:37 smithi038 ceph-mon[28043]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-22T01:06:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:37 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi111 2024-04-22T01:06:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:37 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-22T01:06:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:37 smithi038 ceph-mon[28043]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T01:06:38.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:38.029+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f91ec062670 con 0x7f91d80a2c20 2024-04-22T01:06:38.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:38 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:06:38.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:38 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:06:38.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:38 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:38.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:38 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:06:38.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:38 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:06:38.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:38 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:38.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:38 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:06:38.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:38 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:06:38.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:38 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:39.725 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:39 smithi203 ceph-mon[31825]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:39.726 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:39 smithi203 ceph-mon[31825]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:06:39.806 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:39 smithi038 ceph-mon[28043]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:39.806 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:39 smithi038 ceph-mon[28043]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:06:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:39 smithi111 ceph-mon[31057]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:39 smithi111 ceph-mon[31057]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/265903039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]: dispatch 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]: dispatch 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]': finished 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:41 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/3354863020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:06:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/265903039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]: dispatch 2024-04-22T01:06:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]: dispatch 2024-04-22T01:06:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]': finished 2024-04-22T01:06:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T01:06:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:41.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:41 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/3354863020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/265903039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]: dispatch 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]: dispatch 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e16f8a4-4174-4264-a7ff-976293ac88c1"}]': finished 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:41 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/3354863020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:06:43.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:43 smithi203 ceph-mon[31825]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:43.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:43 smithi111 ceph-mon[31057]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:43.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:43 smithi038 ceph-mon[28043]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:45 smithi203 ceph-mon[31825]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:45.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:45 smithi111 ceph-mon[31057]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:45.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:45 smithi038 ceph-mon[28043]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:47.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:47 smithi203 ceph-mon[31825]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:47 smithi111 ceph-mon[31057]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:47 smithi038 ceph-mon[28043]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:48.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:48 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T01:06:48.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:48 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:48.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:48 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T01:06:48.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:48 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:48.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:48 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T01:06:48.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:48 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:06:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:49 smithi203 ceph-mon[31825]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:49 smithi203 ceph-mon[31825]: Deploying daemon osd.5 on smithi203 2024-04-22T01:06:49.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:49 smithi111 ceph-mon[31057]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:49.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:49 smithi111 ceph-mon[31057]: Deploying daemon osd.5 on smithi203 2024-04-22T01:06:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:49 smithi038 ceph-mon[28043]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:49 smithi038 ceph-mon[28043]: Deploying daemon osd.5 on smithi203 2024-04-22T01:06:51.742 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:51 smithi203 ceph-mon[31825]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:51.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:51 smithi111 ceph-mon[31057]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:51.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:51 smithi038 ceph-mon[28043]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:53.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:53 smithi203 ceph-mon[31825]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:53.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:53 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:53.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:53 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:53.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:53 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:53 smithi111 ceph-mon[31057]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:53 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:53 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:53 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:53.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:53 smithi038 ceph-mon[28043]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:53.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:53.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:53.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:53 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:06:55.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:55 smithi203 ceph-mon[31825]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:55 smithi111 ceph-mon[31057]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:55 smithi038 ceph-mon[28043]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:56.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.797+0000 7f91e2ffd700 1 -- 172.21.15.203:0/340880874 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f91ec062670 con 0x7f91d80a2c20 2024-04-22T01:06:56.798 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d80a2c20 msgr2=0x7f91d80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d80a2c20 0x7f91d80a50e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f91e8005fd0 tx=0x7f91e8005f00 comp rx=0 tx=0).stop 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 msgr2=0x7f91ec1a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 0x7f91ec1a1b70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91e400ada0 tx=0x7f91e400c330 comp rx=0 tx=0).stop 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 shutdown_connections 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d80a2c20 0x7f91d80a50e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec102d50 0x7f91ec1a1b70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ec1011b0 0x7f91ec06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 --2- 172.21.15.203:0/340880874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ec1023b0 0x7f91ec19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:06:56.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 >> 172.21.15.203:0/340880874 conn(0x7f91ec0fc790 msgr2=0x7f91ec105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:06:56.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 shutdown_connections 2024-04-22T01:06:56.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:06:56.800+0000 7f91f40ae700 1 -- 172.21.15.203:0/340880874 wait complete. 2024-04-22T01:06:57.432 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T01:06:57.435 INFO:tasks.cephadm:Deploying osd.6 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-22T01:06:57.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T01:06:57.694 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:06:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:57 smithi203 ceph-mon[31825]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:57 smithi203 ceph-mon[31825]: from='osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T01:06:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:57 smithi203 ceph-mon[31825]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T01:06:57.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:57 smithi038 ceph-mon[28043]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:57.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:57.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:57.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:57 smithi038 ceph-mon[28043]: from='osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T01:06:57.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:57 smithi038 ceph-mon[28043]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T01:06:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:57 smithi111 ceph-mon[31057]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:57 smithi111 ceph-mon[31057]: from='osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T01:06:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:57 smithi111 ceph-mon[31057]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: from='osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:58.803 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-mon[31825]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:06:58.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:58.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:58.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T01:06:58.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: from='osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:06:58.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T01:06:58.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:58.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:58 smithi111 ceph-mon[31057]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: from='osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:58 smithi038 ceph-mon[28043]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:06:59.094 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:06:58.802+0000 7fdac8545700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T01:06:59.094 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:06:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:06:58.812+0000 7fdac4338700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:06:59.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:59 smithi203 ceph-mon[31825]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:59.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:59 smithi203 ceph-mon[31825]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:06:59.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:59 smithi203 ceph-mon[31825]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T01:06:59.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:59 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:59.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:06:59 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:59 smithi111 ceph-mon[31057]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:59 smithi111 ceph-mon[31057]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:06:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:59 smithi111 ceph-mon[31057]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T01:06:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:59 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:06:59 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:59 smithi038 ceph-mon[28043]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:06:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:59 smithi038 ceph-mon[28043]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:06:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:59 smithi038 ceph-mon[28043]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T01:06:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:59 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:06:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:06:59 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:07:00.110 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T01:07:00.110 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T01:07:00.111 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T01:07:00.111 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T01:07:00.111 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155836 s, 673 MB/s 2024-04-22T01:07:00.111 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T01:07:00.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] boot 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:07:00.886 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:00 smithi203 ceph-mon[31825]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T01:07:01.038 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:07:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:07:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:07:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:07:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] boot 2024-04-22T01:07:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T01:07:01.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:07:01.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:00 smithi111 ceph-mon[31057]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T01:07:01.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-22T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: osd.5 [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] boot 2024-04-22T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:00 smithi038 ceph-mon[28043]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi203 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:01 smithi203 ceph-mon[31825]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T01:07:02.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi203 2024-04-22T01:07:02.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:01 smithi038 ceph-mon[28043]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T01:07:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi203 2024-04-22T01:07:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:02.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:01 smithi111 ceph-mon[31057]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 -- 172.21.15.38:0/125229334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8108040 msgr2=0x7f4bf810a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 --2- 172.21.15.38:0/125229334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8108040 0x7f4bf810a430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000b0a0 tx=0x7f4bf000b3b0 comp rx=0 tx=0).stop 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 -- 172.21.15.38:0/125229334 shutdown_connections 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 --2- 172.21.15.38:0/125229334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8108040 0x7f4bf810a430 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 --2- 172.21.15.38:0/125229334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8102de0 0x7f4bf81051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 --2- 172.21.15.38:0/125229334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8105710 0x7f4bf8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 -- 172.21.15.38:0/125229334 >> 172.21.15.38:0/125229334 conn(0x7f4bf80fc730 msgr2=0x7f4bf80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 -- 172.21.15.38:0/125229334 shutdown_connections 2024-04-22T01:07:02.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.950+0000 7f4bfc885700 1 -- 172.21.15.38:0/125229334 wait complete. 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.951+0000 7f4bfc885700 1 Processor -- start 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.951+0000 7f4bfc885700 1 -- start start 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.951+0000 7f4bfc885700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 0x7f4bf81026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8105710 0x7f4bf8100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8108040 0x7f4bf81012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf810deb0 con 0x7f4bf8102de0 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf810dd30 con 0x7f4bf8108040 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf810e030 con 0x7f4bf8105710 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 0x7f4bf81026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 0x7f4bf81026d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38066/0 (socket says 172.21.15.38:38066) 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 -- 172.21.15.38:0/1619479243 learned_addr learned my addr 172.21.15.38:0/1619479243 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:07:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf6d9d700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8108040 0x7f4bf81012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf5d9b700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8105710 0x7f4bf8100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 -- 172.21.15.38:0/1619479243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8105710 msgr2=0x7f4bf8100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8105710 0x7f4bf8100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 -- 172.21.15.38:0/1619479243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8108040 msgr2=0x7f4bf81012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8108040 0x7f4bf81012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 -- 172.21.15.38:0/1619479243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf8101b10 con 0x7f4bf8102de0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf5d9b700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8105710 0x7f4bf8100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf6d9d700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8108040 0x7f4bf81012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bf659c700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 0x7f4bf81026d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4be800b530 tx=0x7f4be800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8015020 con 0x7f4bf8102de0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4be8011d10 con 0x7f4bf8102de0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf81a2540 con 0x7f4bf8102de0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8010e50 con 0x7f4bf8102de0 2024-04-22T01:07:02.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.952+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf81a2a80 con 0x7f4bf8102de0 2024-04-22T01:07:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.953+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf8068f20 con 0x7f4bf8102de0 2024-04-22T01:07:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.955+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4be8018020 con 0x7f4bf8102de0 2024-04-22T01:07:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.955+0000 7f4be77fe700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4be00a2dc0 0x7f4be00a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.955+0000 7f4bf5d9b700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4be00a2dc0 0x7f4be00a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.955+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f4be80cda10 con 0x7f4bf8102de0 2024-04-22T01:07:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.956+0000 7f4bf5d9b700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4be00a2dc0 0x7f4be00a5280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4bec005fd0 tx=0x7f4bec005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:07:02.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:02.960+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4be801f020 con 0x7f4bf8102de0 2024-04-22T01:07:03.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:03.177+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4bf8062670 con 0x7f4be00a2dc0 2024-04-22T01:07:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:03 smithi111 ceph-mon[31057]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:03 smithi111 ceph-mon[31057]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T01:07:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:03.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:03 smithi038 ceph-mon[28043]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:03.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:03 smithi038 ceph-mon[28043]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T01:07:03.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:03.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:03.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:03 smithi203 ceph-mon[31825]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:03 smithi203 ceph-mon[31825]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T01:07:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:03.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:04.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:04 smithi038 ceph-mon[28043]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:04.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:04 smithi038 ceph-mon[28043]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T01:07:04.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:04 smithi203 ceph-mon[31825]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:04.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:04 smithi203 ceph-mon[31825]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T01:07:04.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:04 smithi111 ceph-mon[31057]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:04.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:04 smithi111 ceph-mon[31057]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T01:07:05.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:05 smithi038 ceph-mon[28043]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:05.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:05 smithi203 ceph-mon[31825]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:05 smithi111 ceph-mon[31057]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:06.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/569012313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed061345-8a22-457d-a3dc-13d2ee757388"}]: dispatch 2024-04-22T01:07:06.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/569012313' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed061345-8a22-457d-a3dc-13d2ee757388"}]': finished 2024-04-22T01:07:06.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:06 smithi038 ceph-mon[28043]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T01:07:06.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:06.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:06 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/569012313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed061345-8a22-457d-a3dc-13d2ee757388"}]: dispatch 2024-04-22T01:07:06.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:06 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/569012313' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed061345-8a22-457d-a3dc-13d2ee757388"}]': finished 2024-04-22T01:07:06.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:06 smithi203 ceph-mon[31825]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T01:07:06.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:06.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:06 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/569012313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed061345-8a22-457d-a3dc-13d2ee757388"}]: dispatch 2024-04-22T01:07:06.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:06 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/569012313' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed061345-8a22-457d-a3dc-13d2ee757388"}]': finished 2024-04-22T01:07:06.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:06 smithi111 ceph-mon[31057]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T01:07:06.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:07.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:07 smithi203 ceph-mon[31825]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-22T01:07:07.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:07 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1780272086' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:07.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:07 smithi111 ceph-mon[31057]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-22T01:07:07.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:07 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1780272086' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:07.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:07 smithi038 ceph-mon[28043]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-22T01:07:07.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:07 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1780272086' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:09.682 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:09 smithi038 ceph-mon[28043]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:07:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:09 smithi203 ceph-mon[31825]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:07:09.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:09 smithi111 ceph-mon[31057]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:07:11.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:11 smithi038 ceph-mon[28043]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-22T01:07:11.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:11 smithi203 ceph-mon[31825]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-22T01:07:11.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:11 smithi111 ceph-mon[31057]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-22T01:07:13.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:13 smithi203 ceph-mon[31825]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-22T01:07:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:13 smithi111 ceph-mon[31057]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-22T01:07:13.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:13 smithi038 ceph-mon[28043]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-22T01:07:14.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T01:07:14.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:14.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:14 smithi038 ceph-mon[28043]: Deploying daemon osd.6 on smithi038 2024-04-22T01:07:14.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T01:07:14.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:14.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:14 smithi203 ceph-mon[31825]: Deploying daemon osd.6 on smithi038 2024-04-22T01:07:14.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T01:07:14.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:14.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:14 smithi111 ceph-mon[31057]: Deploying daemon osd.6 on smithi038 2024-04-22T01:07:15.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:15 smithi038 ceph-mon[28043]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:07:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:15 smithi203 ceph-mon[31825]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:07:15.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:15 smithi111 ceph-mon[31057]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:07:17.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:17 smithi038 ceph-mon[28043]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:17 smithi203 ceph-mon[31825]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:17 smithi111 ceph-mon[31057]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:19.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:19 smithi038 ceph-mon[28043]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:19.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:19 smithi203 ceph-mon[31825]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:19.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:19 smithi111 ceph-mon[31057]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:21.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:20 smithi203 ceph-mon[31825]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:21.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:21.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:21.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:21.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:20 smithi111 ceph-mon[31057]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:21.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:21.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:21.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:21.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:20 smithi038 ceph-mon[28043]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:21.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:21.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:21.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:23.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:22 smithi203 ceph-mon[31825]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:23.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:22 smithi038 ceph-mon[28043]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:23.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:22 smithi111 ceph-mon[31057]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:23.539 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 6 on host 'smithi038' 2024-04-22T01:07:23.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.538+0000 7f4be77fe700 1 -- 172.21.15.38:0/1619479243 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4bf8062670 con 0x7f4be00a2dc0 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.540+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4be00a2dc0 msgr2=0x7f4be00a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.540+0000 7f4bfc885700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4be00a2dc0 0x7f4be00a5280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4bec005fd0 tx=0x7f4bec005e20 comp rx=0 tx=0).stop 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.540+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 msgr2=0x7f4bf81026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.540+0000 7f4bfc885700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 0x7f4bf81026d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4be800b530 tx=0x7f4be800b8f0 comp rx=0 tx=0).stop 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 shutdown_connections 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4be00a2dc0 0x7f4be00a5280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf8108040 0x7f4bf81012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bf8102de0 0x7f4bf81026d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 --2- 172.21.15.38:0/1619479243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bf8105710 0x7f4bf8100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 >> 172.21.15.38:0/1619479243 conn(0x7f4bf80fc730 msgr2=0x7f4bf8103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 shutdown_connections 2024-04-22T01:07:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:07:23.541+0000 7f4bfc885700 1 -- 172.21.15.38:0/1619479243 wait complete. 2024-04-22T01:07:24.371 DEBUG:teuthology.orchestra.run.smithi038:osd.6> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T01:07:24.373 INFO:tasks.cephadm:Deploying osd.7 on smithi111 with /dev/vg_nvme/lv_2... 2024-04-22T01:07:24.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T01:07:24.628 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:07:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:24 smithi038 ceph-mon[28043]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T01:07:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.749 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.750 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.750 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:24 smithi111 ceph-mon[31057]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T01:07:24.750 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:24.751 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:24 smithi203 ceph-mon[31825]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T01:07:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:25.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:25 smithi038 ceph-mon[28043]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:25.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:25 smithi038 ceph-mon[28043]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T01:07:25.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:25 smithi038 ceph-mon[28043]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T01:07:25.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:25 smithi038 ceph-mon[28043]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:07:25.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:25 smithi111 ceph-mon[31057]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:25 smithi111 ceph-mon[31057]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T01:07:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:25 smithi111 ceph-mon[31057]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T01:07:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:25 smithi111 ceph-mon[31057]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:07:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:26.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:25 smithi203 ceph-mon[31825]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:26.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:25 smithi203 ceph-mon[31825]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T01:07:26.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:25 smithi203 ceph-mon[31825]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T01:07:26.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:25 smithi203 ceph-mon[31825]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:07:26.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:26.155 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:07:26.015+0000 7f98e29e4700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T01:07:26.155 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:07:26.026+0000 7f98da7cf700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:07:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T01:07:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T01:07:26.862 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-22T01:07:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-22T01:07:26.862 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111618 s, 939 MB/s 2024-04-22T01:07:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-22T01:07:26.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:26 smithi111 ceph-mon[31057]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:07:26.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:26 smithi111 ceph-mon[31057]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T01:07:26.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:26 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:26.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:26 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:26.863 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:26 smithi111 ceph-mon[31057]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' 2024-04-22T01:07:26.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-mon[28043]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:07:26.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-mon[28043]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T01:07:26.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:26.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:26.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:26 smithi038 ceph-mon[28043]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' 2024-04-22T01:07:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:26 smithi203 ceph-mon[31825]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:07:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:26 smithi203 ceph-mon[31825]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T01:07:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:26 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:26 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:26 smithi203 ceph-mon[31825]: from='osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243]' entity='osd.6' 2024-04-22T01:07:27.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-04-22T01:07:27.796 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:07:27.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:07:27.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:07:27.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:27.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.836 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] boot 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:27.837 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] boot 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T01:07:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:27.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:07:27.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: osd.6 [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] boot 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:27.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:27.919 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:27.919 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:28 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi038 2024-04-22T01:07:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:28 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-22T01:07:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:28.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:28 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi038 2024-04-22T01:07:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:28 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-22T01:07:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:28 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi038 2024-04-22T01:07:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:28 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-22T01:07:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 -- 172.21.15.111:0/1190272640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101410 msgr2=0x7fe1e8101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 --2- 172.21.15.111:0/1190272640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101410 0x7fe1e8101890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009990 tx=0x7fe1d8009ca0 comp rx=0 tx=0).stop 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 -- 172.21.15.111:0/1190272640 shutdown_connections 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 --2- 172.21.15.111:0/1190272640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e8101dd0 0x7fe1e8109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 --2- 172.21.15.111:0/1190272640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1e8102840 0x7fe1e8102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 --2- 172.21.15.111:0/1190272640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101410 0x7fe1e8101890 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 -- 172.21.15.111:0/1190272640 >> 172.21.15.111:0/1190272640 conn(0x7fe1e80fbee0 msgr2=0x7fe1e80fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:07:29.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 -- 172.21.15.111:0/1190272640 shutdown_connections 2024-04-22T01:07:29.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.540+0000 7fe1eceae700 1 -- 172.21.15.111:0/1190272640 wait complete. 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 Processor -- start 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 -- start start 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e8101410 0x7fe1e8100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 0x7fe1e819d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1e8102840 0x7fe1e819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810cd80 con 0x7fe1e8102840 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810cc00 con 0x7fe1e8101dd0 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1eceae700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810cf00 con 0x7fe1e8101410 2024-04-22T01:07:29.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1e5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 0x7fe1e819d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1e5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 0x7fe1e819d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55482/0 (socket says 172.21.15.111:55482) 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1e5d9b700 1 -- 172.21.15.111:0/2391224896 learned_addr learned my addr 172.21.15.111:0/2391224896 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.541+0000 7fe1e659c700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e8101410 0x7fe1e8100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1e5d9b700 1 -- 172.21.15.111:0/2391224896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e8101410 msgr2=0x7fe1e8100e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1e5d9b700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e8101410 0x7fe1e8100e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1e5d9b700 1 -- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1e8102840 msgr2=0x7fe1e819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1e5d9b700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1e8102840 0x7fe1e819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1e5d9b700 1 -- 172.21.15.111:0/2391224896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e81a0010 con 0x7fe1e8101dd0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1e5d9b700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 0x7fe1e819d100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005910 tx=0x7fe1d8009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8027070 con 0x7fe1e8101dd0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1d8004150 con 0x7fe1e8101dd0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e81a02a0 con 0x7fe1e8101dd0 2024-04-22T01:07:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d800f460 con 0x7fe1e8101dd0 2024-04-22T01:07:29.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.542+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e81a06f0 con 0x7fe1e8101dd0 2024-04-22T01:07:29.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.543+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e802db30 con 0x7fe1e8101dd0 2024-04-22T01:07:29.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.544+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe1d802a070 con 0x7fe1e8101dd0 2024-04-22T01:07:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.544+0000 7fe1d77fe700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe1d00a2c20 0x7fe1d00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.545+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fe1d80cf9d0 con 0x7fe1e8101dd0 2024-04-22T01:07:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.545+0000 7fe1e659c700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe1d00a2c20 0x7fe1d00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:29.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.545+0000 7fe1e659c700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe1d00a2c20 0x7fe1d00a50e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00ac50 tx=0x7fe1dc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:07:29.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.548+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe1d8031070 con 0x7fe1e8101dd0 2024-04-22T01:07:29.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:29.751+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe1e8062670 con 0x7fe1d00a2c20 2024-04-22T01:07:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:29 smithi111 ceph-mon[31057]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:29 smithi111 ceph-mon[31057]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T01:07:29.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:29 smithi038 ceph-mon[28043]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:29.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:29 smithi038 ceph-mon[28043]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T01:07:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:29 smithi203 ceph-mon[31825]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:29 smithi203 ceph-mon[31825]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T01:07:30.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:30.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:30.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:31 smithi111 ceph-mon[31057]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:31 smithi111 ceph-mon[31057]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:31.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:31 smithi038 ceph-mon[28043]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:31 smithi038 ceph-mon[28043]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:31 smithi203 ceph-mon[31825]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:31 smithi203 ceph-mon[31825]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:32.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:32 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/2880899695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]: dispatch 2024-04-22T01:07:32.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:32 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]: dispatch 2024-04-22T01:07:32.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:32 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]': finished 2024-04-22T01:07:32.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:32 smithi111 ceph-mon[31057]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T01:07:32.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:32 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/2880899695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]: dispatch 2024-04-22T01:07:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:32 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]: dispatch 2024-04-22T01:07:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:32 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]': finished 2024-04-22T01:07:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:32 smithi038 ceph-mon[28043]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T01:07:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:32 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/2880899695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]: dispatch 2024-04-22T01:07:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:32 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]: dispatch 2024-04-22T01:07:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:32 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9832e0c-b8f2-42d4-a384-a43975ec432a"}]': finished 2024-04-22T01:07:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:32 smithi203 ceph-mon[31825]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T01:07:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:33 smithi111 ceph-mon[31057]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:33 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/3174400802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:33.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:33 smithi038 ceph-mon[28043]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:33.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:33 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/3174400802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:33 smithi203 ceph-mon[31825]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:33 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/3174400802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:35 smithi111 ceph-mon[31057]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:35 smithi038 ceph-mon[28043]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:35 smithi203 ceph-mon[31825]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:37.877 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:37 smithi111 ceph-mon[31057]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:37.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:37 smithi038 ceph-mon[28043]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:37 smithi203 ceph-mon[31825]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:39.802 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:39 smithi111 ceph-mon[31057]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:39.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:39 smithi038 ceph-mon[28043]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:40.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:39 smithi203 ceph-mon[31825]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:40.799 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T01:07:40.799 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:40.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T01:07:40.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:41.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T01:07:41.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:41 smithi111 ceph-mon[31057]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:41 smithi111 ceph-mon[31057]: Deploying daemon osd.7 on smithi111 2024-04-22T01:07:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:41 smithi038 ceph-mon[28043]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:41 smithi038 ceph-mon[28043]: Deploying daemon osd.7 on smithi111 2024-04-22T01:07:42.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:41 smithi203 ceph-mon[31825]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:42.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:41 smithi203 ceph-mon[31825]: Deploying daemon osd.7 on smithi111 2024-04-22T01:07:43.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:43 smithi111 ceph-mon[31057]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:43.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:43 smithi038 ceph-mon[28043]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:44.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:43 smithi203 ceph-mon[31825]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:45.775 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:45 smithi111 ceph-mon[31057]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:45.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:45 smithi038 ceph-mon[28043]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:45 smithi203 ceph-mon[31825]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:47.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:47 smithi111 ceph-mon[31057]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:47.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:47.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:47.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:47.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:47 smithi038 ceph-mon[28043]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:47.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:47.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:47.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:47.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:47 smithi203 ceph-mon[31825]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:47.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:47.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:47.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:49.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:49 smithi111 ceph-mon[31057]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:49.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:49 smithi038 ceph-mon[28043]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:49.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:49 smithi203 ceph-mon[31825]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:50.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.073+0000 7fe1d77fe700 1 -- 172.21.15.111:0/2391224896 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe1e8062670 con 0x7fe1d00a2c20 2024-04-22T01:07:50.074 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 7 on host 'smithi111' 2024-04-22T01:07:50.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe1d00a2c20 msgr2=0x7fe1d00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:50.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe1d00a2c20 0x7fe1d00a50e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00ac50 tx=0x7fe1dc00a380 comp rx=0 tx=0).stop 2024-04-22T01:07:50.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 msgr2=0x7fe1e819d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:50.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 0x7fe1e819d100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005910 tx=0x7fe1d8009580 comp rx=0 tx=0).stop 2024-04-22T01:07:50.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 shutdown_connections 2024-04-22T01:07:50.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe1d00a2c20 0x7fe1d00a50e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1e8102840 0x7fe1e819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1e8101410 0x7fe1e8100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 --2- 172.21.15.111:0/2391224896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1e8101dd0 0x7fe1e819d100 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 >> 172.21.15.111:0/2391224896 conn(0x7fe1e80fbee0 msgr2=0x7fe1e8105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:07:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 shutdown_connections 2024-04-22T01:07:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:07:50.076+0000 7fe1eceae700 1 -- 172.21.15.111:0/2391224896 wait complete. 2024-04-22T01:07:50.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.389 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.390 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:50 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:50 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:50.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:50.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:50 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:07:50.712 DEBUG:teuthology.orchestra.run.smithi111:osd.7> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T01:07:50.714 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-04-22T01:07:50.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T01:07:50.962 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:07:51.377 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:51 smithi203 ceph-mon[31825]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:51.377 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:51 smithi203 ceph-mon[31825]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T01:07:51.377 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:51 smithi203 ceph-mon[31825]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T01:07:51.377 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:51 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:51.378 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:51 smithi203 ceph-mon[31825]: from='osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:07:51.378 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:51 smithi203 ceph-mon[31825]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:07:51.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-mon[31057]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:51.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-mon[31057]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T01:07:51.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-mon[31057]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T01:07:51.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:51.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-mon[31057]: from='osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:07:51.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-mon[31057]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:07:51.396 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:07:51.191+0000 7ffb097d6700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T01:07:51.396 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:07:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:07:51.203+0000 7ffb055c9700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:07:51.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:51 smithi038 ceph-mon[28043]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:51.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:51 smithi038 ceph-mon[28043]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T01:07:51.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:51 smithi038 ceph-mon[28043]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T01:07:51.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:51 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:51.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:51 smithi038 ceph-mon[28043]: from='osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:07:51.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:51 smithi038 ceph-mon[28043]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:07:52.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:52 smithi203 ceph-mon[31825]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:07:52.406 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:52 smithi203 ceph-mon[31825]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T01:07:52.406 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:52.406 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:52 smithi203 ceph-mon[31825]: osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] boot 2024-04-22T01:07:52.406 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:52 smithi203 ceph-mon[31825]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T01:07:52.406 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:52.475 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:52 smithi111 ceph-mon[31057]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:07:52.475 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:52 smithi111 ceph-mon[31057]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T01:07:52.476 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:52.476 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:52 smithi111 ceph-mon[31057]: osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] boot 2024-04-22T01:07:52.476 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:52 smithi111 ceph-mon[31057]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T01:07:52.476 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:52.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:52 smithi038 ceph-mon[28043]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:07:52.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:52 smithi038 ceph-mon[28043]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T01:07:52.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:52.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:52 smithi038 ceph-mon[28043]: osd.7 [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] boot 2024-04-22T01:07:52.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:52 smithi038 ceph-mon[28043]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T01:07:52.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T01:07:53.101 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T01:07:53.101 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T01:07:53.101 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-22T01:07:53.101 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-22T01:07:53.102 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120882 s, 867 MB/s 2024-04-22T01:07:53.102 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-22T01:07:53.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:53 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:07:53.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:53 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:07:53.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:53 smithi111 ceph-mon[31057]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:53.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:53 smithi111 ceph-mon[31057]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T01:07:53.397 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:53 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:07:53.398 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:53 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:07:53.398 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:53 smithi203 ceph-mon[31825]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:53.398 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:53 smithi203 ceph-mon[31825]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T01:07:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:53 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:07:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:53 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:07:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:53 smithi038 ceph-mon[28043]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T01:07:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:53 smithi038 ceph-mon[28043]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T01:07:53.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-04-22T01:07:54.004 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:07:55.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:55.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi111 2024-04-22T01:07:55.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.122 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:54 smithi203 ceph-mon[31825]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T01:07:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi111 2024-04-22T01:07:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:54 smithi111 ceph-mon[31057]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T01:07:55.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:55.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi111 2024-04-22T01:07:55.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:07:55.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:54 smithi038 ceph-mon[28043]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 -- 172.21.15.203:0/505382389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103490 msgr2=0x7fbf901038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/505382389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103490 0x7fbf901038f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009990 tx=0x7fbf84009ca0 comp rx=0 tx=0).stop 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 -- 172.21.15.203:0/505382389 shutdown_connections 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/505382389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103e30 0x7fbf9010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/505382389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf90102290 0x7fbf901026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/505382389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103490 0x7fbf901038f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 -- 172.21.15.203:0/505382389 >> 172.21.15.203:0/505382389 conn(0x7fbf900fd830 msgr2=0x7fbf900ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 -- 172.21.15.203:0/505382389 shutdown_connections 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 -- 172.21.15.203:0/505382389 wait complete. 2024-04-22T01:07:55.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.637+0000 7fbf94f3a700 1 Processor -- start 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 -- start start 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf90102290 0x7fbf901984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103490 0x7fbf90198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 0x7fbf9019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9010ded0 con 0x7fbf90102290 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9010dd50 con 0x7fbf90103490 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf94f3a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9010e050 con 0x7fbf90103e30 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 0x7fbf9019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 0x7fbf9019faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:34492/0 (socket says 172.21.15.203:34492) 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 -- 172.21.15.203:0/2874210775 learned_addr learned my addr 172.21.15.203:0/2874210775 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8e59c700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf90102290 0x7fbf901984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8dd9b700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103490 0x7fbf90198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 -- 172.21.15.203:0/2874210775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103490 msgr2=0x7fbf90198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103490 0x7fbf90198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:55.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 -- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf90102290 msgr2=0x7fbf901984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf90102290 0x7fbf901984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 -- 172.21.15.203:0/2874210775 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf901a0140 con 0x7fbf90103e30 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8dd9b700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103490 0x7fbf90198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf8ed9d700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 0x7fbf9019faa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b9a0 tx=0x7fbf8800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.638+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf88019070 con 0x7fbf90103e30 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.639+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf8800ce10 con 0x7fbf90103e30 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.639+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf880073d0 con 0x7fbf90103e30 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.639+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf901a0430 con 0x7fbf90103e30 2024-04-22T01:07:55.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.639+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf901a09f0 con 0x7fbf90103e30 2024-04-22T01:07:55.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.639+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf90068f20 con 0x7fbf90103e30 2024-04-22T01:07:55.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.641+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbf8801c070 con 0x7fbf90103e30 2024-04-22T01:07:55.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.642+0000 7fbf7f7fe700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbf780a2ca0 0x7fbf780a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:07:55.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.642+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7fbf880cdcb0 con 0x7fbf90103e30 2024-04-22T01:07:55.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.642+0000 7fbf8e59c700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbf780a2ca0 0x7fbf780a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:07:55.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.643+0000 7fbf8e59c700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbf780a2ca0 0x7fbf780a5160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf901032f0 tx=0x7fbf8000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:07:55.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.646+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbf8809b770 con 0x7fbf90103e30 2024-04-22T01:07:55.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:07:55.854+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbf90062670 con 0x7fbf780a2ca0 2024-04-22T01:07:56.492 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:56 smithi203 ceph-mon[31825]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T01:07:56.492 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:56.492 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:56.492 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:56 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:56.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:56 smithi111 ceph-mon[31057]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T01:07:56.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:56.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:56.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:56 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:56 smithi038 ceph-mon[28043]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T01:07:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:07:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:07:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:56 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:07:57.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:57 smithi203 ceph-mon[31825]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:57.495 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:57 smithi203 ceph-mon[31825]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:57.495 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:57 smithi203 ceph-mon[31825]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T01:07:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:57 smithi111 ceph-mon[31057]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:57 smithi111 ceph-mon[31057]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:57 smithi111 ceph-mon[31057]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T01:07:57.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:57 smithi038 ceph-mon[28043]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:57 smithi038 ceph-mon[28043]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:57 smithi038 ceph-mon[28043]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T01:07:59.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:59.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/1047480115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]: dispatch 2024-04-22T01:07:59.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]: dispatch 2024-04-22T01:07:59.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]': finished 2024-04-22T01:07:59.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T01:07:59.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:07:59.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:07:59 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/1873908222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:59.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:59.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/1047480115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]: dispatch 2024-04-22T01:07:59.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]: dispatch 2024-04-22T01:07:59.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]': finished 2024-04-22T01:07:59.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T01:07:59.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:07:59.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:07:59 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/1873908222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:07:59.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/1047480115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]: dispatch 2024-04-22T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]: dispatch 2024-04-22T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a5470d-0f1c-45dc-a692-cd4144caf225"}]': finished 2024-04-22T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:07:59 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/1873908222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:01 smithi111 ceph-mon[31057]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:01.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:01 smithi203 ceph-mon[31825]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:01.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:01 smithi038 ceph-mon[28043]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:03.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:03 smithi203 ceph-mon[31825]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:03 smithi111 ceph-mon[31057]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:03.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:03 smithi038 ceph-mon[28043]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:05.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:05 smithi203 ceph-mon[31825]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:05.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:05 smithi111 ceph-mon[31057]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:05.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:05 smithi038 ceph-mon[28043]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:06.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T01:08:06.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:06.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T01:08:06.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:06.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T01:08:06.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:07 smithi203 ceph-mon[31825]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:07 smithi203 ceph-mon[31825]: Deploying daemon osd.8 on smithi203 2024-04-22T01:08:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:07 smithi111 ceph-mon[31057]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:07 smithi111 ceph-mon[31057]: Deploying daemon osd.8 on smithi203 2024-04-22T01:08:07.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:07 smithi038 ceph-mon[28043]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:07 smithi038 ceph-mon[28043]: Deploying daemon osd.8 on smithi203 2024-04-22T01:08:09.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:09 smithi203 ceph-mon[31825]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:09.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:09 smithi038 ceph-mon[28043]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:09 smithi111 ceph-mon[31057]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:11.376 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:11 smithi203 ceph-mon[31825]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:11.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:11 smithi111 ceph-mon[31057]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:11.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:11 smithi038 ceph-mon[28043]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:12.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:12.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:12.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:13.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:13 smithi203 ceph-mon[31825]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:13 smithi111 ceph-mon[31057]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:13 smithi038 ceph-mon[28043]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:15.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.524+0000 7fbf7f7fe700 1 -- 172.21.15.203:0/2874210775 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbf90062670 con 0x7fbf780a2ca0 2024-04-22T01:08:15.525 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbf780a2ca0 msgr2=0x7fbf780a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbf780a2ca0 0x7fbf780a5160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf901032f0 tx=0x7fbf8000a380 comp rx=0 tx=0).stop 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 msgr2=0x7fbf9019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 0x7fbf9019faa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b9a0 tx=0x7fbf8800bcb0 comp rx=0 tx=0).stop 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 shutdown_connections 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbf780a2ca0 0x7fbf780a5160 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf90103e30 0x7fbf9019faa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf90102290 0x7fbf901984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 --2- 172.21.15.203:0/2874210775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf90103490 0x7fbf90198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:15.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 >> 172.21.15.203:0/2874210775 conn(0x7fbf900fd830 msgr2=0x7fbf90106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:08:15.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 shutdown_connections 2024-04-22T01:08:15.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:08:15.527+0000 7fbf94f3a700 1 -- 172.21.15.203:0/2874210775 wait complete. 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T01:08:16.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T01:08:16.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:16.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T01:08:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T01:08:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:16.182 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.8.service 2024-04-22T01:08:16.184 INFO:tasks.cephadm:Deploying osd.9 on smithi038 with /dev/vg_nvme/lv_1... 2024-04-22T01:08:16.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T01:08:16.441 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:08:16.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:16 smithi203 ceph-mon[31825]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T01:08:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:16 smithi203 ceph-mon[31825]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T01:08:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:16 smithi203 ceph-mon[31825]: from='osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:08:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:16 smithi203 ceph-mon[31825]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:08:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:16 smithi203 ceph-mon[31825]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:16.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:16 smithi038 ceph-mon[28043]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T01:08:16.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:16 smithi038 ceph-mon[28043]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T01:08:16.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:16.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:16 smithi038 ceph-mon[28043]: from='osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:08:16.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:16 smithi038 ceph-mon[28043]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:08:16.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:16 smithi038 ceph-mon[28043]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:17.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:16 smithi111 ceph-mon[31057]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T01:08:17.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:16 smithi111 ceph-mon[31057]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T01:08:17.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:17.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:16 smithi111 ceph-mon[31057]: from='osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:08:17.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:16 smithi111 ceph-mon[31057]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:08:17.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:16 smithi111 ceph-mon[31057]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:17.516 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:08:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:08:17.183+0000 7f5b3d326700 -1 osd.8 0 waiting for initial osdmap 2024-04-22T01:08:17.516 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:08:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:08:17.194+0000 7f5b35111700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:08:17.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:17 smithi203 ceph-mon[31825]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:08:17.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:17 smithi203 ceph-mon[31825]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T01:08:17.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:17.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:17 smithi203 ceph-mon[31825]: from='osd.8 ' entity='osd.8' 2024-04-22T01:08:17.853 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:17 smithi038 ceph-mon[28043]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:08:17.854 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:17 smithi038 ceph-mon[28043]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T01:08:17.854 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:17.854 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:17 smithi038 ceph-mon[28043]: from='osd.8 ' entity='osd.8' 2024-04-22T01:08:18.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:17 smithi111 ceph-mon[31057]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:08:18.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:17 smithi111 ceph-mon[31057]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T01:08:18.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:18.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:17 smithi111 ceph-mon[31057]: from='osd.8 ' entity='osd.8' 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0154694 s, 678 MB/s 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 2024-04-22T01:08:18.723 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] boot 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:18.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:18 smithi203 ceph-mon[31825]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T01:08:19.112 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:08:19.113 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:08:19.113 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] boot 2024-04-22T01:08:19.113 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T01:08:19.113 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:19.113 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:19.113 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:18 smithi038 ceph-mon[28043]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T01:08:19.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:08:19.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:08:19.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: osd.8 [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] boot 2024-04-22T01:08:19.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T01:08:19.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T01:08:19.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-22T01:08:19.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:18 smithi111 ceph-mon[31057]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T01:08:19.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-04-22T01:08:19.815 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:08:20.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi203 2024-04-22T01:08:20.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:20 smithi111 ceph-mon[31057]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi203 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:20.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:20.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:20 smithi038 ceph-mon[28043]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi203 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:20.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:20.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:20 smithi203 ceph-mon[31825]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T01:08:21.592 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:21 smithi038 ceph-mon[28043]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:21.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3508847348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc0102d50 msgr2=0x7fdcc0109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3508847348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc0102d50 0x7fdcc0109600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800b0d0 tx=0x7fdcb800b3e0 comp rx=0 tx=0).stop 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3508847348 shutdown_connections 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3508847348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc0102d50 0x7fdcc0109600 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3508847348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc01011b0 0x7fdcc01015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3508847348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01023b0 0x7fdcc0102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3508847348 >> 172.21.15.38:0/3508847348 conn(0x7fdcc00fc750 msgr2=0x7fdcc00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3508847348 shutdown_connections 2024-04-22T01:08:21.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.601+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3508847348 wait complete. 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 Processor -- start 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 -- start start 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01011b0 0x7fdcc01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 0x7fdcc0198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc0102d50 0x7fdcc019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010ce90 con 0x7fdcc01023b0 2024-04-22T01:08:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010cd10 con 0x7fdcc01011b0 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcc4a07700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010d010 con 0x7fdcc0102d50 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcbdd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 0x7fdcc0198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcbdd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 0x7fdcc0198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36066/0 (socket says 172.21.15.38:36066) 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.602+0000 7fdcbdd9b700 1 -- 172.21.15.38:0/3451155121 learned_addr learned my addr 172.21.15.38:0/3451155121 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbe59c700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01011b0 0x7fdcc01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbed9d700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc0102d50 0x7fdcc019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbdd9b700 1 -- 172.21.15.38:0/3451155121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc0102d50 msgr2=0x7fdcc019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbdd9b700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc0102d50 0x7fdcc019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbdd9b700 1 -- 172.21.15.38:0/3451155121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01011b0 msgr2=0x7fdcc01985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbdd9b700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01011b0 0x7fdcc01985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbdd9b700 1 -- 172.21.15.38:0/3451155121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc01a0230 con 0x7fdcc01023b0 2024-04-22T01:08:21.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcbdd9b700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 0x7fdcc0198b10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b560 tx=0x7fdcb400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:08:21.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4015020 con 0x7fdcc01023b0 2024-04-22T01:08:21.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc01a0520 con 0x7fdcc01023b0 2024-04-22T01:08:21.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcb4011d10 con 0x7fdcc01023b0 2024-04-22T01:08:21.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb400f2e0 con 0x7fdcc01023b0 2024-04-22T01:08:21.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.603+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc01a0b30 con 0x7fdcc01023b0 2024-04-22T01:08:21.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.604+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc0068f20 con 0x7fdcc01023b0 2024-04-22T01:08:21.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.605+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdcb4018020 con 0x7fdcc01023b0 2024-04-22T01:08:21.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.606+0000 7fdcaf7fe700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fdca80a2ca0 0x7fdca80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:21.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.606+0000 7fdcbe59c700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fdca80a2ca0 0x7fdca80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:21.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.606+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7fdcb40ce3d0 con 0x7fdcc01023b0 2024-04-22T01:08:21.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.606+0000 7fdcbe59c700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fdca80a2ca0 0x7fdca80a5160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0006fd0 tx=0x7fdcb0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:08:21.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.610+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdcb401f020 con 0x7fdcc01023b0 2024-04-22T01:08:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:21 smithi111 ceph-mon[31057]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:21 smithi203 ceph-mon[31825]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:21.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:21.824+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdcc00fe8e0 con 0x7fdca80a2ca0 2024-04-22T01:08:22.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:22 smithi038 ceph-mon[28043]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T01:08:22.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:08:22.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:08:22.610 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:22 smithi111 ceph-mon[31057]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T01:08:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:08:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:08:22.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:22 smithi203 ceph-mon[31825]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T01:08:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:08:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:08:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:23.612 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:23 smithi038 ceph-mon[28043]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:23.612 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:23 smithi038 ceph-mon[28043]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:08:23.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:23 smithi038 ceph-mon[28043]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T01:08:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:23 smithi111 ceph-mon[31057]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:23 smithi111 ceph-mon[31057]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:08:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:23 smithi111 ceph-mon[31057]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T01:08:23.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:23 smithi203 ceph-mon[31825]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:23.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:23 smithi203 ceph-mon[31825]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:08:23.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:23 smithi203 ceph-mon[31825]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T01:08:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:25 smithi111 ceph-mon[31057]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-22T01:08:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:25 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3331005974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d125ca4b-b70c-471e-864b-6eb587daf211"}]: dispatch 2024-04-22T01:08:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:25 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3331005974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d125ca4b-b70c-471e-864b-6eb587daf211"}]': finished 2024-04-22T01:08:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:25 smithi111 ceph-mon[31057]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T01:08:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:25 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:25 smithi038 ceph-mon[28043]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-22T01:08:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:25 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3331005974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d125ca4b-b70c-471e-864b-6eb587daf211"}]: dispatch 2024-04-22T01:08:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:25 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3331005974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d125ca4b-b70c-471e-864b-6eb587daf211"}]': finished 2024-04-22T01:08:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:25 smithi038 ceph-mon[28043]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T01:08:25.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:25 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:25 smithi203 ceph-mon[31825]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-22T01:08:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:25 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3331005974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d125ca4b-b70c-471e-864b-6eb587daf211"}]: dispatch 2024-04-22T01:08:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:25 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3331005974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d125ca4b-b70c-471e-864b-6eb587daf211"}]': finished 2024-04-22T01:08:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:25 smithi203 ceph-mon[31825]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T01:08:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:25 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:26.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:26 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/235823436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:26.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:26 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/235823436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:26.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:26 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/235823436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:27.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:27 smithi111 ceph-mon[31057]: pgmap v209: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:08:27.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:27 smithi038 ceph-mon[28043]: pgmap v209: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:08:27.667 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:27 smithi203 ceph-mon[31825]: pgmap v209: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:08:29.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:29 smithi111 ceph-mon[31057]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T01:08:29.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:29 smithi038 ceph-mon[28043]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T01:08:29.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:29 smithi203 ceph-mon[31825]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T01:08:31.545 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:31 smithi038 ceph-mon[28043]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T01:08:31.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:31 smithi111 ceph-mon[31057]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T01:08:31.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:31 smithi203 ceph-mon[31825]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T01:08:33.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:33 smithi111 ceph-mon[31057]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:33.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T01:08:33.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:33.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:33 smithi038 ceph-mon[28043]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:33.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T01:08:33.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:33.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:33 smithi203 ceph-mon[31825]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:33.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T01:08:33.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:34.662 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:34 smithi038 ceph-mon[28043]: Deploying daemon osd.9 on smithi038 2024-04-22T01:08:34.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:34 smithi203 ceph-mon[31825]: Deploying daemon osd.9 on smithi038 2024-04-22T01:08:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:34 smithi111 ceph-mon[31057]: Deploying daemon osd.9 on smithi038 2024-04-22T01:08:35.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:35 smithi038 ceph-mon[28043]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:35.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:35 smithi203 ceph-mon[31825]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:35 smithi111 ceph-mon[31057]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:37.635 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:37 smithi038 ceph-mon[28043]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:37.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:37 smithi203 ceph-mon[31825]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:37 smithi111 ceph-mon[31057]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:39 smithi203 ceph-mon[31825]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:39 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:39 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:39 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:39.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:39 smithi038 ceph-mon[28043]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:39.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:39 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:39.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:39 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:39.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:39 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:39 smithi111 ceph-mon[31057]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:39 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:39 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:39 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:41.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:41 smithi038 ceph-mon[28043]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:41 smithi203 ceph-mon[31825]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:41 smithi111 ceph-mon[31057]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:43.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.013+0000 7fdcaf7fe700 1 -- 172.21.15.38:0/3451155121 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdcc00fe8e0 con 0x7fdca80a2ca0 2024-04-22T01:08:43.014 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 9 on host 'smithi038' 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.015+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fdca80a2ca0 msgr2=0x7fdca80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.015+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fdca80a2ca0 0x7fdca80a5160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0006fd0 tx=0x7fdcb0008040 comp rx=0 tx=0).stop 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.015+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 msgr2=0x7fdcc0198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.015+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 0x7fdcc0198b10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b560 tx=0x7fdcb400b920 comp rx=0 tx=0).stop 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.015+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 shutdown_connections 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fdca80a2ca0 0x7fdca80a5160 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc0102d50 0x7fdcc019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc01011b0 0x7fdcc01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 --2- 172.21.15.38:0/3451155121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdcc01023b0 0x7fdcc0198b10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 >> 172.21.15.38:0/3451155121 conn(0x7fdcc00fc750 msgr2=0x7fdcc00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:08:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 shutdown_connections 2024-04-22T01:08:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:08:43.016+0000 7fdcc4a07700 1 -- 172.21.15.38:0/3451155121 wait complete. 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:43 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:43 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T01:08:43.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:43.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:43 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:08:43.811 DEBUG:teuthology.orchestra.run.smithi038:osd.9> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.9.service 2024-04-22T01:08:43.813 INFO:tasks.cephadm:Deploying osd.10 on smithi111 with /dev/vg_nvme/lv_1... 2024-04-22T01:08:43.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T01:08:44.059 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:08:44.556 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:44 smithi111 ceph-mon[31057]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T01:08:44.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:44 smithi111 ceph-mon[31057]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T01:08:44.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:44 smithi111 ceph-mon[31057]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:08:44.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:44 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:44 smithi038 ceph-mon[28043]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T01:08:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:44 smithi038 ceph-mon[28043]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T01:08:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:44 smithi038 ceph-mon[28043]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:08:44.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:44 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:44.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:08:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:08:44.443+0000 7f9ae70f6700 -1 osd.9 0 waiting for initial osdmap 2024-04-22T01:08:44.656 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:08:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:08:44.459+0000 7f9ae2ee9700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:08:44.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:44 smithi203 ceph-mon[31825]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T01:08:44.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:44 smithi203 ceph-mon[31825]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T01:08:44.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:44 smithi203 ceph-mon[31825]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-22T01:08:44.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:44 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:45.542 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:45 smithi111 ceph-mon[31057]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:45.542 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:45 smithi111 ceph-mon[31057]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:08:45.542 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:45 smithi111 ceph-mon[31057]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T01:08:45.543 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:45.543 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:45 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:45.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:45 smithi038 ceph-mon[28043]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:45.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:45 smithi038 ceph-mon[28043]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:08:45.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:45 smithi038 ceph-mon[28043]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T01:08:45.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:45.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:45 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:45 smithi203 ceph-mon[31825]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:45 smithi203 ceph-mon[31825]: from='osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-22T01:08:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:45 smithi203 ceph-mon[31825]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T01:08:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:45 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T01:08:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T01:08:46.279 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-22T01:08:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-22T01:08:46.279 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110909 s, 945 MB/s 2024-04-22T01:08:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-22T01:08:46.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:46 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:08:46.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:46 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:08:46.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:46 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:46 smithi111 ceph-mon[31057]: osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] boot 2024-04-22T01:08:46.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:46 smithi111 ceph-mon[31057]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T01:08:46.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:46 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:46 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:08:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:46 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:08:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:46 smithi038 ceph-mon[28043]: osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] boot 2024-04-22T01:08:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:46 smithi038 ceph-mon[28043]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T01:08:46.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:46 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:46 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:08:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:46 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:08:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:46 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:46 smithi203 ceph-mon[31825]: osd.9 [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] boot 2024-04-22T01:08:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:46 smithi203 ceph-mon[31825]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T01:08:46.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:46 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T01:08:46.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-04-22T01:08:47.229 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:08:47.632 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:47.632 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:47.633 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:47 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:47.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:47.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:47 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-22T01:08:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T01:08:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:08:47.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:47 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:08:48.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:48 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi038 2024-04-22T01:08:48.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:48 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-22T01:08:48.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:48 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi038 2024-04-22T01:08:48.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:48 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-22T01:08:48.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2814837632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4101190 msgr2=0x7fd6d41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:48.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2814837632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4101190 0x7fd6d41015b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40099c0 tx=0x7fd6c4009cd0 comp rx=0 tx=0).stop 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2814837632 shutdown_connections 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2814837632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102d30 0x7fd6d41095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2814837632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4101190 0x7fd6d41015b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2814837632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102390 0x7fd6d41027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2814837632 >> 172.21.15.111:0/2814837632 conn(0x7fd6d40fc730 msgr2=0x7fd6d40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.859+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2814837632 shutdown_connections 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.860+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2814837632 wait complete. 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.860+0000 7fd6d99fb700 1 Processor -- start 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.860+0000 7fd6d99fb700 1 -- start start 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.860+0000 7fd6d99fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101190 0x7fd6d4105120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.860+0000 7fd6d99fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 0x7fd6d4103770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.860+0000 7fd6d99fb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102d30 0x7fd6d4103cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d99fb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410cdd0 con 0x7fd6d4101190 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d99fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410cc50 con 0x7fd6d4102390 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d99fb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410cf50 con 0x7fd6d4102d30 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 0x7fd6d4103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 0x7fd6d4103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45408/0 (socket says 172.21.15.111:45408) 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 -- 172.21.15.111:0/2739308374 learned_addr learned my addr 172.21.15.111:0/2739308374 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d2ffd700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101190 0x7fd6d4105120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d37fe700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102d30 0x7fd6d4103cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 -- 172.21.15.111:0/2739308374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102d30 msgr2=0x7fd6d4103cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102d30 0x7fd6d4103cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 -- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101190 msgr2=0x7fd6d4105120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101190 0x7fd6d4105120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 -- 172.21.15.111:0/2739308374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d4104560 con 0x7fd6d4102390 2024-04-22T01:08:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d37fe700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102d30 0x7fd6d4103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:08:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.861+0000 7fd6d27fc700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 0x7fd6d4103770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800c920 tx=0x7fd6c800cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:08:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.862+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8007b00 con 0x7fd6d4102390 2024-04-22T01:08:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.862+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d41a2700 con 0x7fd6d4102390 2024-04-22T01:08:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.862+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c8004d10 con 0x7fd6d4102390 2024-04-22T01:08:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.862+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8005630 con 0x7fd6d4102390 2024-04-22T01:08:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.862+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d41a2cc0 con 0x7fd6d4102390 2024-04-22T01:08:48.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.863+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d4068f20 con 0x7fd6d4102390 2024-04-22T01:08:48.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.864+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6c8004750 con 0x7fd6d4102390 2024-04-22T01:08:48.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.864+0000 7fd6d89f9700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6bc0a2c20 0x7fd6bc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:08:48.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.865+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fd6c80ce810 con 0x7fd6d4102390 2024-04-22T01:08:48.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.865+0000 7fd6d2ffd700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6bc0a2c20 0x7fd6bc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:08:48.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.865+0000 7fd6d2ffd700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6bc0a2c20 0x7fd6bc0a50e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c00 tx=0x7fd6c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:08:48.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:48.868+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd6c8014030 con 0x7fd6d4102390 2024-04-22T01:08:48.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:48 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi038 2024-04-22T01:08:48.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:48 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-22T01:08:49.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:08:49.071+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd6d40fe940 con 0x7fd6bc0a2c20 2024-04-22T01:08:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:49 smithi203 ceph-mon[31825]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:08:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:08:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:49 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:49.824 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:49 smithi111 ceph-mon[31057]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:49.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:08:49.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:08:49.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:49 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:49 smithi038 ceph-mon[28043]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:08:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:08:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:49 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:08:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:50 smithi203 ceph-mon[31825]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:08:50.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:50 smithi111 ceph-mon[31057]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:08:50.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:50 smithi038 ceph-mon[28043]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:08:51.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:51 smithi203 ceph-mon[31825]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:51.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:51 smithi111 ceph-mon[31057]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:51.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:51 smithi038 ceph-mon[28043]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:52.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:52 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/4057949825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]: dispatch 2024-04-22T01:08:52.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:52 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]: dispatch 2024-04-22T01:08:52.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:52 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]': finished 2024-04-22T01:08:52.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:52 smithi111 ceph-mon[31057]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T01:08:52.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:08:52.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:52 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/4057949825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]: dispatch 2024-04-22T01:08:52.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:52 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]: dispatch 2024-04-22T01:08:52.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:52 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]': finished 2024-04-22T01:08:52.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:52 smithi203 ceph-mon[31825]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T01:08:52.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:08:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:52 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/4057949825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]: dispatch 2024-04-22T01:08:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:52 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]: dispatch 2024-04-22T01:08:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:52 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36"}]': finished 2024-04-22T01:08:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:52 smithi038 ceph-mon[28043]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T01:08:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:08:53.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:53 smithi203 ceph-mon[31825]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:53.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.111:0/727955560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:53 smithi111 ceph-mon[31057]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.111:0/727955560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:53.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:53 smithi038 ceph-mon[28043]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:53.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.111:0/727955560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:08:55.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:55 smithi203 ceph-mon[31825]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:55.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:55 smithi111 ceph-mon[31057]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:55 smithi038 ceph-mon[28043]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:57 smithi203 ceph-mon[31825]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:57 smithi111 ceph-mon[31057]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:57 smithi038 ceph-mon[28043]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:59.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:59 smithi203 ceph-mon[31825]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:59.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:08:59 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T01:08:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:59 smithi111 ceph-mon[31057]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:08:59 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T01:08:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:59 smithi038 ceph-mon[28043]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:08:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:08:59 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T01:09:00.705 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:00.705 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:00 smithi111 ceph-mon[31057]: Deploying daemon osd.10 on smithi111 2024-04-22T01:09:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:00 smithi038 ceph-mon[28043]: Deploying daemon osd.10 on smithi111 2024-04-22T01:09:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:00 smithi203 ceph-mon[31825]: Deploying daemon osd.10 on smithi111 2024-04-22T01:09:01.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:01 smithi111 ceph-mon[31057]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:01 smithi038 ceph-mon[28043]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:01 smithi203 ceph-mon[31825]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:03.638 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:03 smithi111 ceph-mon[31057]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:03.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:03 smithi038 ceph-mon[28043]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:04.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:03 smithi203 ceph-mon[31825]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:05 smithi111 ceph-mon[31057]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:05 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:05 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:05 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:05 smithi038 ceph-mon[28043]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:05 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:05 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:05 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:05 smithi203 ceph-mon[31825]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:05 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:05 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:05 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:07.839 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:07 smithi111 ceph-mon[31057]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:07.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:07 smithi038 ceph-mon[28043]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:07 smithi203 ceph-mon[31825]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:09.392 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 10 on host 'smithi111' 2024-04-22T01:09:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.393+0000 7fd6d89f9700 1 -- 172.21.15.111:0/2739308374 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd6d40fe940 con 0x7fd6bc0a2c20 2024-04-22T01:09:09.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6bc0a2c20 msgr2=0x7fd6bc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6bc0a2c20 0x7fd6bc0a50e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c00 tx=0x7fd6c4005fb0 comp rx=0 tx=0).stop 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 msgr2=0x7fd6d4103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 0x7fd6d4103770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800c920 tx=0x7fd6c800cc30 comp rx=0 tx=0).stop 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 shutdown_connections 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6bc0a2c20 0x7fd6bc0a50e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102d30 0x7fd6d4103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101190 0x7fd6d4105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 --2- 172.21.15.111:0/2739308374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4102390 0x7fd6d4103770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 >> 172.21.15.111:0/2739308374 conn(0x7fd6d40fc730 msgr2=0x7fd6d40fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 shutdown_connections 2024-04-22T01:09:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-22T01:09:09.395+0000 7fd6d99fb700 1 -- 172.21.15.111:0/2739308374 wait complete. 2024-04-22T01:09:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.706 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:09.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:09.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:09.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:10.122 DEBUG:teuthology.orchestra.run.smithi111:osd.10> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.10.service 2024-04-22T01:09:10.124 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-04-22T01:09:10.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T01:09:10.372 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:09:10.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:10 smithi203 ceph-mon[31825]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T01:09:10.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:10 smithi203 ceph-mon[31825]: from='osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:09:10.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:10 smithi203 ceph-mon[31825]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T01:09:10.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:10.753 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:10 smithi203 ceph-mon[31825]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:09:10.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-mon[31057]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T01:09:10.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-mon[31057]: from='osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:09:10.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-mon[31057]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T01:09:10.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:10.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-mon[31057]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:09:10.896 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:09:10.556+0000 7fc9fe3cc700 -1 osd.10 0 waiting for initial osdmap 2024-04-22T01:09:10.896 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:09:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:09:10.569+0000 7fc9fa1bf700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:09:10.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:10 smithi038 ceph-mon[28043]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T01:09:10.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:10 smithi038 ceph-mon[28043]: from='osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:09:10.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:10 smithi038 ceph-mon[28043]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T01:09:10.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:10.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:10 smithi038 ceph-mon[28043]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-22T01:09:11.746 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:11.746 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:09:11.746 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T01:09:11.746 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.747 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.747 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] boot 2024-04-22T01:09:11.747 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T01:09:11.747 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] boot 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T01:09:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: osd.10 [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] boot 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T01:09:11.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T01:09:12.679 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T01:09:12.679 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T01:09:12.679 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-22T01:09:12.679 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-22T01:09:12.679 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124677 s, 841 MB/s 2024-04-22T01:09:12.679 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-22T01:09:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:12 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:09:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:12 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:09:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:12 smithi203 ceph-mon[31825]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T01:09:12.888 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:12 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:09:12.888 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:12 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:09:12.888 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:12 smithi111 ceph-mon[31057]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T01:09:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:12 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:09:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:12 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:09:12.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:12 smithi038 ceph-mon[28043]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T01:09:13.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-04-22T01:09:13.590 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:09:13.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:13.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:13.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:13.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:13 smithi203 ceph-mon[31825]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T01:09:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:13.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:13.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:13 smithi111 ceph-mon[31057]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:13.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:13.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:13.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:13 smithi038 ceph-mon[28043]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T01:09:14.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:14 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi111 2024-04-22T01:09:14.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:14 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-22T01:09:14.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:14 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi111 2024-04-22T01:09:14.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:14 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-22T01:09:14.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:14 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi111 2024-04-22T01:09:14.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:14 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- 172.21.15.203:0/3332538989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 msgr2=0x7fb574101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- 172.21.15.203:0/3332538989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb574101fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb560009990 tx=0x7fb560009ca0 comp rx=0 tx=0).stop 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- 172.21.15.203:0/3332538989 shutdown_connections 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- 172.21.15.203:0/3332538989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5741024e0 0x7fb574108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- 172.21.15.203:0/3332538989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb574100940 0x7fb574100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- 172.21.15.203:0/3332538989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb574101fa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- 172.21.15.203:0/3332538989 >> 172.21.15.203:0/3332538989 conn(0x7fb5740fbee0 msgr2=0x7fb5740fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- 172.21.15.203:0/3332538989 shutdown_connections 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- 172.21.15.203:0/3332538989 wait complete. 2024-04-22T01:09:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 Processor -- start 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- start start 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb574100940 0x7fb57410a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb57419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5741024e0 0x7fb57419b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb57410cea0 con 0x7fb5741024e0 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb57410cd20 con 0x7fb574100940 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.339+0000 7fb57ae98700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb57410d020 con 0x7fb574101b40 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb57419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb57419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:60090/0 (socket says 172.21.15.203:60090) 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 -- 172.21.15.203:0/952605346 learned_addr learned my addr 172.21.15.203:0/952605346 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb579435700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5741024e0 0x7fb57419b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:15.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb578c34700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb574100940 0x7fb57410a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 -- 172.21.15.203:0/952605346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb574100940 msgr2=0x7fb57410a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb574100940 0x7fb57410a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 -- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5741024e0 msgr2=0x7fb57419b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5741024e0 0x7fb57419b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 -- 172.21.15.203:0/952605346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57419c020 con 0x7fb574101b40 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb573fff700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb57419b0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb560005910 tx=0x7fb5600047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560027070 con 0x7fb574101b40 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb560004210 con 0x7fb574101b40 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57419c2b0 con 0x7fb574101b40 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56000f460 con 0x7fb574101b40 2024-04-22T01:09:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.340+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57419c810 con 0x7fb574101b40 2024-04-22T01:09:15.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.342+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb574068f20 con 0x7fb574101b40 2024-04-22T01:09:15.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.343+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb56002a070 con 0x7fb574101b40 2024-04-22T01:09:15.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.343+0000 7fb571ffb700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb5640a2c20 0x7fb5640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:15.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.343+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7fb5600cfd40 con 0x7fb574101b40 2024-04-22T01:09:15.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.343+0000 7fb578c34700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb5640a2c20 0x7fb5640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:15.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.344+0000 7fb578c34700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb5640a2c20 0x7fb5640a50e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5741019a0 tx=0x7fb568009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:15.350 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.347+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb560013070 con 0x7fb574101b40 2024-04-22T01:09:15.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:15.556+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb574107950 con 0x7fb5640a2c20 2024-04-22T01:09:15.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:15 smithi111 ceph-mon[31057]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:15.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:15 smithi111 ceph-mon[31057]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T01:09:15.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:09:15.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:09:15.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:15 smithi038 ceph-mon[28043]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:15 smithi038 ceph-mon[28043]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T01:09:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:09:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:09:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:15 smithi203 ceph-mon[31825]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:15 smithi203 ceph-mon[31825]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T01:09:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T01:09:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T01:09:16.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:16 smithi203 ceph-mon[31825]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:09:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:16 smithi203 ceph-mon[31825]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T01:09:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:16 smithi111 ceph-mon[31057]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:09:16.936 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:16 smithi111 ceph-mon[31057]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T01:09:16.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:16 smithi038 ceph-mon[28043]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:09:16.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:16 smithi038 ceph-mon[28043]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T01:09:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:17 smithi203 ceph-mon[31825]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:17 smithi111 ceph-mon[31057]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:17 smithi038 ceph-mon[28043]: pgmap v246: 1 pgs: 1 remapped+peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:18 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/1500087181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]: dispatch 2024-04-22T01:09:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:18 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]: dispatch 2024-04-22T01:09:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:18 smithi111 ceph-mon[31057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]': finished 2024-04-22T01:09:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:18 smithi111 ceph-mon[31057]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T01:09:18.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:18 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/1500087181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]: dispatch 2024-04-22T01:09:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:18 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]: dispatch 2024-04-22T01:09:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:18 smithi038 ceph-mon[28043]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]': finished 2024-04-22T01:09:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:18 smithi038 ceph-mon[28043]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T01:09:18.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:19.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:18 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/1500087181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]: dispatch 2024-04-22T01:09:19.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:18 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]: dispatch 2024-04-22T01:09:19.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:18 smithi203 ceph-mon[31825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60ce13a3-6d11-48e8-b53e-609c433deaf3"}]': finished 2024-04-22T01:09:19.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:18 smithi203 ceph-mon[31825]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T01:09:19.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:18 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:19.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:19 smithi111 ceph-mon[31057]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-22T01:09:19.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:19 smithi111 ceph-mon[31057]: from='client.? 172.21.15.203:0/1907728977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:09:19.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:19 smithi038 ceph-mon[28043]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-22T01:09:19.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:19 smithi038 ceph-mon[28043]: from='client.? 172.21.15.203:0/1907728977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:09:20.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:19 smithi203 ceph-mon[31825]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-22T01:09:20.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:19 smithi203 ceph-mon[31825]: from='client.? 172.21.15.203:0/1907728977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T01:09:21.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:21 smithi111 ceph-mon[31057]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:09:21.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:21 smithi038 ceph-mon[28043]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:09:21.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:21 smithi203 ceph-mon[31825]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T01:09:23.764 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:23 smithi203 ceph-mon[31825]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-22T01:09:23.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:23 smithi111 ceph-mon[31057]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-22T01:09:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:23 smithi038 ceph-mon[28043]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-22T01:09:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:25 smithi111 ceph-mon[31057]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T01:09:25.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:25 smithi038 ceph-mon[28043]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T01:09:26.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:25 smithi203 ceph-mon[31825]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T01:09:26.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:26 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T01:09:26.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:26 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:26.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:26 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T01:09:26.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:26 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:26.908 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:26 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T01:09:26.909 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:26 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:27 smithi111 ceph-mon[31057]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:09:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:27 smithi111 ceph-mon[31057]: Deploying daemon osd.11 on smithi203 2024-04-22T01:09:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:27 smithi038 ceph-mon[28043]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:09:27.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:27 smithi038 ceph-mon[28043]: Deploying daemon osd.11 on smithi203 2024-04-22T01:09:27.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:27 smithi203 ceph-mon[31825]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T01:09:27.959 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:27 smithi203 ceph-mon[31825]: Deploying daemon osd.11 on smithi203 2024-04-22T01:09:29.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:29 smithi203 ceph-mon[31825]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:29.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:29 smithi038 ceph-mon[28043]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:30.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:29 smithi111 ceph-mon[31057]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:31.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:31 smithi038 ceph-mon[28043]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:31.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:31 smithi111 ceph-mon[31057]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:31 smithi203 ceph-mon[31825]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:32.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:32 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:33.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:33.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:33.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:32 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:33.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:33.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:33.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:32 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:33.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:33 smithi203 ceph-mon[31825]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:34.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:33 smithi111 ceph-mon[31057]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:34.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:33 smithi038 ceph-mon[28043]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:35.480 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-04-22T01:09:35.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.479+0000 7fb571ffb700 1 -- 172.21.15.203:0/952605346 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb574107950 con 0x7fb5640a2c20 2024-04-22T01:09:35.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb5640a2c20 msgr2=0x7fb5640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:35.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb5640a2c20 0x7fb5640a50e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5741019a0 tx=0x7fb568009190 comp rx=0 tx=0).stop 2024-04-22T01:09:35.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 msgr2=0x7fb57419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:35.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb57419b0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb560005910 tx=0x7fb5600047b0 comp rx=0 tx=0).stop 2024-04-22T01:09:35.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 shutdown_connections 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb5640a2c20 0x7fb5640a50e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5741024e0 0x7fb57419b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb574100940 0x7fb57410a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 --2- 172.21.15.203:0/952605346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb574101b40 0x7fb57419b0d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 >> 172.21.15.203:0/952605346 conn(0x7fb5740fbee0 msgr2=0x7fb574104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 shutdown_connections 2024-04-22T01:09:35.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:35.482+0000 7fb57ae98700 1 -- 172.21.15.203:0/952605346 wait complete. 2024-04-22T01:09:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:35 smithi203 ceph-mon[31825]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:35 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:35 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:35 smithi111 ceph-mon[31057]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:35 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:35 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:35 smithi038 ceph-mon[28043]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:36.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:35 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:35 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.185 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.11.service 2024-04-22T01:09:36.187 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-22T01:09:36.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T01:09:36.447 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:09:36.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:36 smithi038 ceph-mon[28043]: from='osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T01:09:36.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:36 smithi038 ceph-mon[28043]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T01:09:36.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:36 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:36 smithi203 ceph-mon[31825]: from='osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T01:09:36.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:36 smithi203 ceph-mon[31825]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T01:09:36.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:36.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:36 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:37.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:36 smithi111 ceph-mon[31057]: from='osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T01:09:37.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:36 smithi111 ceph-mon[31057]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T01:09:37.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:37.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:36 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:37.837 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:37 smithi038 ceph-mon[28043]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:37.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:37 smithi038 ceph-mon[28043]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T01:09:37.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:37 smithi038 ceph-mon[28043]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T01:09:37.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:37 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:37.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:37 smithi038 ceph-mon[28043]: from='osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:09:37.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:37 smithi038 ceph-mon[28043]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:09:38.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:37 smithi203 ceph-mon[31825]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:38.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:37 smithi203 ceph-mon[31825]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T01:09:38.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:37 smithi203 ceph-mon[31825]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T01:09:38.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:37 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:38.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:37 smithi203 ceph-mon[31825]: from='osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:09:38.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:37 smithi203 ceph-mon[31825]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:09:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:37 smithi111 ceph-mon[31057]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:37 smithi111 ceph-mon[31057]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T01:09:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:37 smithi111 ceph-mon[31057]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T01:09:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:37 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:37 smithi111 ceph-mon[31057]: from='osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:09:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:37 smithi111 ceph-mon[31057]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-22T01:09:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 -- 172.21.15.38:0/1137697048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 msgr2=0x7f3a80105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/1137697048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a80105950 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009990 tx=0x7f3a6c009ca0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 -- 172.21.15.38:0/1137697048 shutdown_connections 2024-04-22T01:09:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/1137697048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a80105e90 0x7f3a80106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/1137697048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a80100b50 0x7f3a80102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/1137697048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a80105950 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 -- 172.21.15.38:0/1137697048 >> 172.21.15.38:0/1137697048 conn(0x7f3a800fc790 msgr2=0x7f3a800febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 -- 172.21.15.38:0/1137697048 shutdown_connections 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 -- 172.21.15.38:0/1137697048 wait complete. 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 Processor -- start 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.219+0000 7f3a86ac6700 1 -- start start 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a80100b50 0x7f3a8019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a8019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a80105e90 0x7f3a801a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3a80114440 con 0x7f3a801034c0 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3a801142c0 con 0x7f3a80105e90 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a801145c0 con 0x7f3a80100b50 2024-04-22T01:09:38.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a8019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a8019eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44870/0 (socket says 172.21.15.38:44870) 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a84862700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a80100b50 0x7f3a8019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 -- 172.21.15.38:0/531635817 learned_addr learned my addr 172.21.15.38:0/531635817 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a85063700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a80105e90 0x7f3a801a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 -- 172.21.15.38:0/531635817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a80100b50 msgr2=0x7f3a8019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a80100b50 0x7f3a8019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 -- 172.21.15.38:0/531635817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a80105e90 msgr2=0x7f3a801a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a80105e90 0x7f3a801a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 -- 172.21.15.38:0/531635817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a801a6600 con 0x7f3a801034c0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7ffff700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a8019eee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00bbb0 tx=0x7f3a6c00bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c026070 con 0x7f3a801034c0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c00f460 con 0x7f3a801034c0 2024-04-22T01:09:38.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a801a6890 con 0x7f3a801034c0 2024-04-22T01:09:38.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.220+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c021650 con 0x7f3a801034c0 2024-04-22T01:09:38.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.221+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a801a6da0 con 0x7f3a801034c0 2024-04-22T01:09:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.222+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a80068f20 con 0x7f3a801034c0 2024-04-22T01:09:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.223+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3a6c029070 con 0x7f3a801034c0 2024-04-22T01:09:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.223+0000 7f3a7dffb700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3a700a2c20 0x7f3a700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.223+0000 7f3a84862700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3a700a2c20 0x7f3a700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.223+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f3a6c0cfca0 con 0x7f3a801034c0 2024-04-22T01:09:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.224+0000 7f3a84862700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3a700a2c20 0x7f3a700a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c60 tx=0x7f3a74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:38.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.227+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3a6c030020 con 0x7f3a801034c0 2024-04-22T01:09:38.361 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:09:38.123+0000 7f3525d52700 -1 osd.11 0 waiting for initial osdmap 2024-04-22T01:09:38.361 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:09:38.137+0000 7f351db3d700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T01:09:38.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.433+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3a80062e40 con 0x7f3a801034c0 2024-04-22T01:09:38.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.433+0000 7f3a7dffb700 1 -- 172.21.15.38:0/531635817 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f3a6c013070 con 0x7f3a801034c0 2024-04-22T01:09:38.433 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:38.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3a700a2c20 msgr2=0x7f3a700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:38.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3a700a2c20 0x7f3a700a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c60 tx=0x7f3a74009260 comp rx=0 tx=0).stop 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 msgr2=0x7f3a8019eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a8019eee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00bbb0 tx=0x7f3a6c00bbe0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 shutdown_connections 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3a700a2c20 0x7f3a700a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a80105e90 0x7f3a801a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a80100b50 0x7f3a8019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 --2- 172.21.15.38:0/531635817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a801034c0 0x7f3a8019eee0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 >> 172.21.15.38:0/531635817 conn(0x7f3a800fc790 msgr2=0x7f3a80101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 shutdown_connections 2024-04-22T01:09:38.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:38.436+0000 7f3a86ac6700 1 -- 172.21.15.38:0/531635817 wait complete. 2024-04-22T01:09:39.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-mon[31825]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:09:39.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-mon[31825]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T01:09:39.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:39.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-mon[31825]: from='osd.11 ' entity='osd.11' 2024-04-22T01:09:39.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/531635817' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T01:09:39.206 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:38 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:39.206 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:38 smithi111 ceph-mon[31057]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:09:39.206 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:38 smithi111 ceph-mon[31057]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:38 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:38 smithi111 ceph-mon[31057]: from='osd.11 ' entity='osd.11' 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:38 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/531635817' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:38 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:38 smithi038 ceph-mon[28043]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:38 smithi038 ceph-mon[28043]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:38 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:38 smithi038 ceph-mon[28043]: from='osd.11 ' entity='osd.11' 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:38 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/531635817' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T01:09:39.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:38 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:39.208 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1713748151,"num_in_osds":12,"osd_in_since":1713748158,"num_remapped_pgs":0} 2024-04-22T01:09:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:39 smithi203 ceph-mon[31825]: purged_snaps scrub starts 2024-04-22T01:09:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:39 smithi203 ceph-mon[31825]: purged_snaps scrub ok 2024-04-22T01:09:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:39 smithi203 ceph-mon[31825]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:39 smithi203 ceph-mon[31825]: osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] boot 2024-04-22T01:09:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:39 smithi203 ceph-mon[31825]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T01:09:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:39 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:39 smithi111 ceph-mon[31057]: purged_snaps scrub starts 2024-04-22T01:09:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:39 smithi111 ceph-mon[31057]: purged_snaps scrub ok 2024-04-22T01:09:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:39 smithi111 ceph-mon[31057]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:39 smithi111 ceph-mon[31057]: osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] boot 2024-04-22T01:09:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:39 smithi111 ceph-mon[31057]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T01:09:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:39 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:40.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:39 smithi038 ceph-mon[28043]: purged_snaps scrub starts 2024-04-22T01:09:40.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:39 smithi038 ceph-mon[28043]: purged_snaps scrub ok 2024-04-22T01:09:40.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:39 smithi038 ceph-mon[28043]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-22T01:09:40.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:39 smithi038 ceph-mon[28043]: osd.11 [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] boot 2024-04-22T01:09:40.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:39 smithi038 ceph-mon[28043]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T01:09:40.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:39 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T01:09:40.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T01:09:40.456 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: Detected new or changed devices on smithi203 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:41.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:41.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:41.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: Detected new or changed devices on smithi203 2024-04-22T01:09:41.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:41.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: Detected new or changed devices on smithi203 2024-04-22T01:09:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:41.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 -- 172.21.15.38:0/191597576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc108000 msgr2=0x7fafcc10a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 --2- 172.21.15.38:0/191597576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc108000 0x7fafcc10a410 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fafc400b0a0 tx=0x7fafc400b3b0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 -- 172.21.15.38:0/191597576 shutdown_connections 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 --2- 172.21.15.38:0/191597576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc108000 0x7fafcc10a410 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 --2- 172.21.15.38:0/191597576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafcc102da0 0x7fafcc105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 --2- 172.21.15.38:0/191597576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc1056d0 0x7fafcc107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 -- 172.21.15.38:0/191597576 >> 172.21.15.38:0/191597576 conn(0x7fafcc0fc790 msgr2=0x7fafcc0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:42.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 -- 172.21.15.38:0/191597576 shutdown_connections 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 -- 172.21.15.38:0/191597576 wait complete. 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 Processor -- start 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.294+0000 7fafd347e700 1 -- start start 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc102da0 0x7fafcc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 0x7fafcc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafcc108000 0x7fafcc19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc10dec0 con 0x7fafcc1056d0 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc10dd40 con 0x7fafcc108000 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc10e040 con 0x7fafcc102da0 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 0x7fafcc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 0x7fafcc198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44888/0 (socket says 172.21.15.38:44888) 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 -- 172.21.15.38:0/2887977814 learned_addr learned my addr 172.21.15.38:0/2887977814 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd1a1b700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafcc108000 0x7fafcc19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:42.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd121a700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc102da0 0x7fafcc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 -- 172.21.15.38:0/2887977814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc102da0 msgr2=0x7fafcc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc102da0 0x7fafcc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 -- 172.21.15.38:0/2887977814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafcc108000 msgr2=0x7fafcc19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafcc108000 0x7fafcc19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 -- 172.21.15.38:0/2887977814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc1a0070 con 0x7fafcc1056d0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd121a700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc102da0 0x7fafcc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd0a19700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 0x7fafcc198950 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fafc800d9b0 tx=0x7fafc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8016070 con 0x7fafcc1056d0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8004d10 con 0x7fafcc1056d0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc1a0360 con 0x7fafcc1056d0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8005630 con 0x7fafcc1056d0 2024-04-22T01:09:42.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.295+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc1a08a0 con 0x7fafcc1056d0 2024-04-22T01:09:42.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.296+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafcc068f20 con 0x7fafcc1056d0 2024-04-22T01:09:42.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.298+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fafc8004750 con 0x7fafcc1056d0 2024-04-22T01:09:42.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.298+0000 7fafc27fc700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fafb80a2c20 0x7fafb80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:42.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.298+0000 7fafd121a700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fafb80a2c20 0x7fafb80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:42.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.298+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fafc80ceee0 con 0x7fafcc1056d0 2024-04-22T01:09:42.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.299+0000 7fafd121a700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fafb80a2c20 0x7fafb80a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:42.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.302+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fafc801f020 con 0x7fafcc1056d0 2024-04-22T01:09:42.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.506+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fafcc02db30 con 0x7fafcc1056d0 2024-04-22T01:09:42.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.507+0000 7fafc27fc700 1 -- 172.21.15.38:0/2887977814 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fafc809c2e0 con 0x7fafcc1056d0 2024-04-22T01:09:42.507 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:42.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fafb80a2c20 msgr2=0x7fafb80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:42.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fafb80a2c20 0x7fafb80a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc005ee0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 msgr2=0x7fafcc198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:42.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 0x7fafcc198950 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fafc800d9b0 tx=0x7fafc800dcc0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 shutdown_connections 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fafb80a2c20 0x7fafb80a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafcc108000 0x7fafcc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc102da0 0x7fafcc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 --2- 172.21.15.38:0/2887977814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafcc1056d0 0x7fafcc198950 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 >> 172.21.15.38:0/2887977814 conn(0x7fafcc0fc790 msgr2=0x7fafcc1036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 shutdown_connections 2024-04-22T01:09:42.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:42.509+0000 7fafd347e700 1 -- 172.21.15.38:0/2887977814 wait complete. 2024-04-22T01:09:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:42 smithi038 ceph-mon[28043]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:43.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:42 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2887977814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T01:09:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:42 smithi203 ceph-mon[31825]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:43.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:42 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2887977814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T01:09:43.324 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713748179,"num_in_osds":12,"osd_in_since":1713748158,"num_remapped_pgs":0} 2024-04-22T01:09:43.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T01:09:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:42 smithi111 ceph-mon[31057]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:42 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2887977814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T01:09:43.575 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:09:45.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:44 smithi038 ceph-mon[28043]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:45.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:44 smithi203 ceph-mon[31825]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:45.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:44 smithi111 ceph-mon[31057]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:45.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 -- 172.21.15.38:0/4257311330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1075a0 msgr2=0x7fb7bc10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:45.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 --2- 172.21.15.38:0/4257311330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc10f170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009990 tx=0x7fb7ac009ca0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 -- 172.21.15.38:0/4257311330 shutdown_connections 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 --2- 172.21.15.38:0/4257311330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc10f170 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 --2- 172.21.15.38:0/4257311330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 --2- 172.21.15.38:0/4257311330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc102e90 0x7fb7bc106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 -- 172.21.15.38:0/4257311330 >> 172.21.15.38:0/4257311330 conn(0x7fb7bc0793e0 msgr2=0x7fb7bc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 -- 172.21.15.38:0/4257311330 shutdown_connections 2024-04-22T01:09:45.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.464+0000 7fb7c4236700 1 -- 172.21.15.38:0/4257311330 wait complete. 2024-04-22T01:09:45.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 Processor -- start 2024-04-22T01:09:45.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 -- start start 2024-04-22T01:09:45.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc19e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:45.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc102e90 0x7fb7bc19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1126b0 con 0x7fb7bc1024e0 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc112530 con 0x7fb7bc1075a0 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c4236700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc112830 con 0x7fb7bc102e90 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc19e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc19e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44916/0 (socket says 172.21.15.38:44916) 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 -- 172.21.15.38:0/1311556138 learned_addr learned my addr 172.21.15.38:0/1311556138 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c17d1700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc102e90 0x7fb7bc19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c27d3700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 -- 172.21.15.38:0/1311556138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc102e90 msgr2=0x7fb7bc19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc102e90 0x7fb7bc19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 -- 172.21.15.38:0/1311556138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1075a0 msgr2=0x7fb7bc198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c1fd2700 1 -- 172.21.15.38:0/1311556138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc1990f0 con 0x7fb7bc1024e0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.465+0000 7fb7c27d3700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.466+0000 7fb7c1fd2700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc19e780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400ba50 tx=0x7fb7b400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.466+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4019070 con 0x7fb7bc1024e0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.466+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b4014440 con 0x7fb7bc1024e0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.466+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc1993e0 con 0x7fb7bc1024e0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.466+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40075d0 con 0x7fb7bc1024e0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.466+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc1a5650 con 0x7fb7bc1024e0 2024-04-22T01:09:45.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.467+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc068f20 con 0x7fb7bc1024e0 2024-04-22T01:09:45.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.468+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb7b401c070 con 0x7fb7bc1024e0 2024-04-22T01:09:45.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.468+0000 7fb7baffd700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb7a40a2c20 0x7fb7a40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:45.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.469+0000 7fb7c17d1700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb7a40a2c20 0x7fb7a40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:45.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.469+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7b40ce030 con 0x7fb7bc1024e0 2024-04-22T01:09:45.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.469+0000 7fb7c17d1700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb7a40a2c20 0x7fb7a40a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006fd0 tx=0x7fb7b0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:45.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.473+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb7b409b430 con 0x7fb7bc1024e0 2024-04-22T01:09:45.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.680+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7bc199e70 con 0x7fb7bc1024e0 2024-04-22T01:09:45.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.681+0000 7fb7baffd700 1 -- 172.21.15.38:0/1311556138 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18491 (secure 0 0 0) 0x7fb7b409ad30 con 0x7fb7bc1024e0 2024-04-22T01:09:45.682 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:45.685 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:09:40.128137+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb7a40a2c20 msgr2=0x7fb7a40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb7a40a2c20 0x7fb7a40a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006fd0 tx=0x7fb7b0008040 comp rx=0 tx=0).stop 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 msgr2=0x7fb7bc19e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc19e780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400ba50 tx=0x7fb7b400be10 comp rx=0 tx=0).stop 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 shutdown_connections 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb7a40a2c20 0x7fb7a40a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc1075a0 0x7fb7bc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb7bc1024e0 0x7fb7bc19e780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 --2- 172.21.15.38:0/1311556138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc102e90 0x7fb7bc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 >> 172.21.15.38:0/1311556138 conn(0x7fb7bc0793e0 msgr2=0x7fb7bc100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.683+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 shutdown_connections 2024-04-22T01:09:45.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:45.684+0000 7fb7c4236700 1 -- 172.21.15.38:0/1311556138 wait complete. 2024-04-22T01:09:46.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:45 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1311556138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:09:46.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:45 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1311556138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:09:46.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:45 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1311556138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:09:46.437 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:09:46.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T01:09:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:09:47.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:46 smithi038 ceph-mon[28043]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:47.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:46 smithi203 ceph-mon[31825]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:47.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:46 smithi111 ceph-mon[31057]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:48.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 -- 172.21.15.38:0/1470691776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 msgr2=0x7fc1f010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:48.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/1470691776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f010f160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800b0d0 tx=0x7fc1e800b3e0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 -- 172.21.15.38:0/1470691776 shutdown_connections 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/1470691776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f010f160 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/1470691776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1f006ac40 0x7fc1f006b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/1470691776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1f006b5f0 0x7fc1f0106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 -- 172.21.15.38:0/1470691776 >> 172.21.15.38:0/1470691776 conn(0x7fc1f00781d0 msgr2=0x7fc1f00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 -- 172.21.15.38:0/1470691776 shutdown_connections 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.320+0000 7fc1f4d25700 1 -- 172.21.15.38:0/1470691776 wait complete. 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 Processor -- start 2024-04-22T01:09:48.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 -- start start 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1f006ac40 0x7fc1f019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1f006b5f0 0x7fc1f019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f0112590 con 0x7fc1f0107590 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f0112410 con 0x7fc1f006ac40 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1f4d25700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f0112710 con 0x7fc1f006b5f0 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f0198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45058/0 (socket says 172.21.15.38:45058) 2024-04-22T01:09:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 -- 172.21.15.38:0/3217919020 learned_addr learned my addr 172.21.15.38:0/3217919020 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1edd9b700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1f006b5f0 0x7fc1f019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1ee59c700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1f006ac40 0x7fc1f019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 -- 172.21.15.38:0/3217919020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1f006b5f0 msgr2=0x7fc1f019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1f006b5f0 0x7fc1f019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 -- 172.21.15.38:0/3217919020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1f006ac40 msgr2=0x7fc1f019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1f006ac40 0x7fc1f019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.321+0000 7fc1eed9d700 1 -- 172.21.15.38:0/3217919020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f0199010 con 0x7fc1f0107590 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.322+0000 7fc1eed9d700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f0198830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8006070 tx=0x7fc1e8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.322+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8021070 con 0x7fc1f0107590 2024-04-22T01:09:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.322+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e8007b80 con 0x7fc1f0107590 2024-04-22T01:09:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.322+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f01992a0 con 0x7fc1f0107590 2024-04-22T01:09:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.322+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8005330 con 0x7fc1f0107590 2024-04-22T01:09:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.322+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f01a54c0 con 0x7fc1f0107590 2024-04-22T01:09:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.323+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f0103990 con 0x7fc1f0107590 2024-04-22T01:09:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.324+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc1e8024070 con 0x7fc1f0107590 2024-04-22T01:09:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.324+0000 7fc1df7fe700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1d80a2ca0 0x7fc1d80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.325+0000 7fc1ee59c700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1d80a2ca0 0x7fc1d80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.325+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc1e800f020 con 0x7fc1f0107590 2024-04-22T01:09:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.325+0000 7fc1ee59c700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1d80a2ca0 0x7fc1d80a5160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0005950 tx=0x7fc1e00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:48.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.329+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc1e802b020 con 0x7fc1f0107590 2024-04-22T01:09:48.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.533+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc1f002db30 con 0x7fc1f0107590 2024-04-22T01:09:48.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.534+0000 7fc1df7fe700 1 -- 172.21.15.38:0/3217919020 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fc1e809d570 con 0x7fc1f0107590 2024-04-22T01:09:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:09:48.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1d80a2ca0 msgr2=0x7fc1d80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:48.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1d80a2ca0 0x7fc1d80a5160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0005950 tx=0x7fc1e00058e0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 msgr2=0x7fc1f0198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f0198830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8006070 tx=0x7fc1e8004d00 comp rx=0 tx=0).stop 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 shutdown_connections 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1d80a2ca0 0x7fc1d80a5160 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1f0107590 0x7fc1f0198830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1f006ac40 0x7fc1f019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 --2- 172.21.15.38:0/3217919020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1f006b5f0 0x7fc1f019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 >> 172.21.15.38:0/3217919020 conn(0x7fc1f00781d0 msgr2=0x7fc1f00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 shutdown_connections 2024-04-22T01:09:48.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:48.536+0000 7fc1f4d25700 1 -- 172.21.15.38:0/3217919020 wait complete. 2024-04-22T01:09:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:48 smithi203 ceph-mon[31825]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:49.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:48 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3217919020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:09:49.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:48 smithi038 ceph-mon[28043]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:49.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:48 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3217919020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:09:49.384 INFO:tasks.cephadm:Adding mds.a on smithi038 2024-04-22T01:09:49.384 INFO:tasks.cephadm:Adding mds.d on smithi038 2024-04-22T01:09:49.384 INFO:tasks.cephadm:Adding mds.g on smithi038 2024-04-22T01:09:49.384 INFO:tasks.cephadm:Adding mds.j on smithi038 2024-04-22T01:09:49.384 INFO:tasks.cephadm:Adding mds.b on smithi111 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.e on smithi111 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.h on smithi111 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.k on smithi111 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-04-22T01:09:49.385 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-04-22T01:09:49.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-04-22T01:09:49.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:48 smithi111 ceph-mon[31057]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:49.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:48 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3217919020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:09:49.634 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:09:51.097 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:50 smithi203 ceph-mon[31825]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 -- 172.21.15.203:0/1463311637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4101190 msgr2=0x7f80f41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 --2- 172.21.15.203:0/1463311637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4101190 0x7f80f41015b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009990 tx=0x7f80f0009ca0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 -- 172.21.15.203:0/1463311637 shutdown_connections 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 --2- 172.21.15.203:0/1463311637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4102d30 0x7f80f41095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 --2- 172.21.15.203:0/1463311637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4101190 0x7f80f41015b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 --2- 172.21.15.203:0/1463311637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102390 0x7f80f41027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 -- 172.21.15.203:0/1463311637 >> 172.21.15.203:0/1463311637 conn(0x7f80f40fc730 msgr2=0x7f80f40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 -- 172.21.15.203:0/1463311637 shutdown_connections 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 -- 172.21.15.203:0/1463311637 wait complete. 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 Processor -- start 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.326+0000 7f80fbc56700 1 -- start start 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4101190 0x7f80f406b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 0x7f80f419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102d30 0x7f80f419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80f410ce50 con 0x7f80f4102d30 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f80f410ccd0 con 0x7f80f4101190 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80f410cfd0 con 0x7f80f4102390 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 0x7f80f419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 0x7f80f419d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:34744/0 (socket says 172.21.15.203:34744) 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 -- 172.21.15.203:0/3608439419 learned_addr learned my addr 172.21.15.203:0/3608439419 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-22T01:09:51.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fa1f3700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102d30 0x7f80f419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f99f2700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4101190 0x7f80f406b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 -- 172.21.15.203:0/3608439419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4101190 msgr2=0x7f80f406b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4101190 0x7f80f406b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 -- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102d30 msgr2=0x7f80f419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102d30 0x7f80f419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 -- 172.21.15.203:0/3608439419 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f41a02e0 con 0x7f80f4102390 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f99f2700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4101190 0x7f80f406b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fa1f3700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102d30 0x7f80f419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80f91f1700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 0x7f80f419d440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f80e400b530 tx=0x7f80e400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4015020 con 0x7f80f4102390 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f41a05d0 con 0x7f80f4102390 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80e4011d10 con 0x7f80f4102390 2024-04-22T01:09:51.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4010e50 con 0x7f80f4102390 2024-04-22T01:09:51.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.327+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f41a0be0 con 0x7f80f4102390 2024-04-22T01:09:51.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.328+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f4068f20 con 0x7f80f4102390 2024-04-22T01:09:51.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.329+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f80e4018020 con 0x7f80f4102390 2024-04-22T01:09:51.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.330+0000 7f80eaffd700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80e00a2dc0 0x7f80e00a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:51.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.330+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f80e40ce940 con 0x7f80f4102390 2024-04-22T01:09:51.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.330+0000 7f80f99f2700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80e00a2dc0 0x7f80e00a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:51.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.331+0000 7f80f99f2700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80e00a2dc0 0x7f80e00a5280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000c00 tx=0x7f80f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:51.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.334+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f80e401f020 con 0x7f80f4102390 2024-04-22T01:09:51.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:50 smithi111 ceph-mon[31057]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:51.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:50 smithi038 ceph-mon[28043]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:51.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.539+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) -- 0x7f80f4062870 con 0x7f80e00a2dc0 2024-04-22T01:09:51.547 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.548+0000 7f80eaffd700 1 -- 172.21.15.203:0/3608439419 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f80f4062870 con 0x7f80e00a2dc0 2024-04-22T01:09:51.548 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80e00a2dc0 msgr2=0x7f80e00a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80e00a2dc0 0x7f80e00a5280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000c00 tx=0x7f80f0005fb0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 msgr2=0x7f80f419d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 0x7f80f419d440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f80e400b530 tx=0x7f80e400b8f0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 shutdown_connections 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80e00a2dc0 0x7f80e00a5280 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80f4102d30 0x7f80f419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f4101190 0x7f80f406b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 --2- 172.21.15.203:0/3608439419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80f4102390 0x7f80f419d440 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:51.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 >> 172.21.15.203:0/3608439419 conn(0x7f80f40fc730 msgr2=0x7f80f4105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:51.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 shutdown_connections 2024-04-22T01:09:51.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-22T01:09:51.550+0000 7f80fbc56700 1 -- 172.21.15.203:0/3608439419 wait complete. 2024-04-22T01:09:52.248 DEBUG:teuthology.orchestra.run.smithi038:mds.a> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.a.service 2024-04-22T01:09:52.251 DEBUG:teuthology.orchestra.run.smithi038:mds.d> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.d.service 2024-04-22T01:09:52.294 DEBUG:teuthology.orchestra.run.smithi038:mds.g> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.g.service 2024-04-22T01:09:52.337 DEBUG:teuthology.orchestra.run.smithi038:mds.j> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.j.service 2024-04-22T01:09:52.381 DEBUG:teuthology.orchestra.run.smithi111:mds.b> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.b.service 2024-04-22T01:09:52.384 DEBUG:teuthology.orchestra.run.smithi111:mds.e> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.e.service 2024-04-22T01:09:52.427 DEBUG:teuthology.orchestra.run.smithi111:mds.h> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.h.service 2024-04-22T01:09:52.470 DEBUG:teuthology.orchestra.run.smithi111:mds.k> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.k.service 2024-04-22T01:09:52.513 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.c.service 2024-04-22T01:09:52.516 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.f.service 2024-04-22T01:09:52.559 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.i.service 2024-04-22T01:09:52.602 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.l.service 2024-04-22T01:09:52.644 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-22T01:09:52.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:09:52.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 -- 172.21.15.38:0/974154568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c1017b0 msgr2=0x7f381c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 --2- 172.21.15.38:0/974154568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c1017b0 0x7f381c101bd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f380c009900 tx=0x7f380c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 -- 172.21.15.38:0/974154568 shutdown_connections 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 --2- 172.21.15.38:0/974154568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f381c103360 0x7f381c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 --2- 172.21.15.38:0/974154568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381c1029a0 0x7f381c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 --2- 172.21.15.38:0/974154568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c1017b0 0x7f381c101bd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 -- 172.21.15.38:0/974154568 >> 172.21.15.38:0/974154568 conn(0x7f381c0fcd20 msgr2=0x7f381c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 -- 172.21.15.38:0/974154568 shutdown_connections 2024-04-22T01:09:52.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.764+0000 7f3820c96640 1 -- 172.21.15.38:0/974154568 wait complete. 2024-04-22T01:09:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 Processor -- start 2024-04-22T01:09:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 -- start start 2024-04-22T01:09:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381c1017b0 0x7f381c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f381c1029a0 0x7f381c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 0x7f381c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10d400 con 0x7f381c103360 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10d280 con 0x7f381c1029a0 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3820c96640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10d580 con 0x7f381c1017b0 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 0x7f381c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 0x7f381c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45074/0 (socket says 172.21.15.38:45074) 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 -- 172.21.15.38:0/1428828644 learned_addr learned my addr 172.21.15.38:0/1428828644 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381a59c640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381c1017b0 0x7f381c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f3819d9b640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f381c1029a0 0x7f381c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 -- 172.21.15.38:0/1428828644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381c1017b0 msgr2=0x7f381c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381c1017b0 0x7f381c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 -- 172.21.15.38:0/1428828644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f381c1029a0 msgr2=0x7f381c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f381c1029a0 0x7f381c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.765+0000 7f381ad9d640 1 -- 172.21.15.38:0/1428828644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c19f720 con 0x7f381c103360 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.766+0000 7f381ad9d640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 0x7f381c19f0b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3814009fd0 tx=0x7f381400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.766+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814017070 con 0x7f381c103360 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.766+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3814003e50 con 0x7f381c103360 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.766+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c19fa10 con 0x7f381c103360 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.766+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38140106c0 con 0x7f381c103360 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.766+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c19ff60 con 0x7f381c103360 2024-04-22T01:09:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.767+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c108ba0 con 0x7f381c103360 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:52.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:52 smithi203 ceph-mon[31825]: Deploying daemon mds.l on smithi203 2024-04-22T01:09:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.768+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3814003970 con 0x7f381c103360 2024-04-22T01:09:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.768+0000 7f380b7fe640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f38040a2c10 0x7f38040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.769+0000 7f381a59c640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f38040a2c10 0x7f38040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.769+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f38140ce790 con 0x7f381c103360 2024-04-22T01:09:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.769+0000 7f381a59c640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f38040a2c10 0x7f38040a50d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f380c000c00 tx=0x7f380c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.773+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f381409b540 con 0x7f381c103360 2024-04-22T01:09:52.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:09:52.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-22T01:09:52.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:52.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:52.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:52 smithi111 ceph-mon[31057]: Deploying daemon mds.l on smithi203 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:52.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:52 smithi038 ceph-mon[28043]: Deploying daemon mds.l on smithi203 2024-04-22T01:09:52.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.962+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f381c062340 con 0x7f381c103360 2024-04-22T01:09:52.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.963+0000 7f380b7fe640 1 -- 172.21.15.38:0/1428828644 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18491 (secure 0 0 0) 0x7f381409f840 con 0x7f381c103360 2024-04-22T01:09:52.964 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:52.964 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:09:40.128137+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f38040a2c10 msgr2=0x7f38040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f38040a2c10 0x7f38040a50d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f380c000c00 tx=0x7f380c005fd0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 msgr2=0x7f381c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 0x7f381c19f0b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3814009fd0 tx=0x7f381400b260 comp rx=0 tx=0).stop 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 shutdown_connections 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f38040a2c10 0x7f38040a50d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f381c103360 0x7f381c19f0b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f381c1029a0 0x7f381c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 --2- 172.21.15.38:0/1428828644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381c1017b0 0x7f381c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 >> 172.21.15.38:0/1428828644 conn(0x7f381c0fcd20 msgr2=0x7f381c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 shutdown_connections 2024-04-22T01:09:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:52.965+0000 7f3820c96640 1 -- 172.21.15.38:0/1428828644 wait complete. 2024-04-22T01:09:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:09:52.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:09:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 -- 172.21.15.38:0/2528502401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 msgr2=0x7fb23c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 --2- 172.21.15.38:0/2528502401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c102e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009900 tx=0x7fb22c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:09:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 -- 172.21.15.38:0/2528502401 shutdown_connections 2024-04-22T01:09:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 --2- 172.21.15.38:0/2528502401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb23c103350 0x7fb23c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 --2- 172.21.15.38:0/2528502401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c102e10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 --2- 172.21.15.38:0/2528502401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb23c1017a0 0x7fb23c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 -- 172.21.15.38:0/2528502401 >> 172.21.15.38:0/2528502401 conn(0x7fb23c0fcd10 msgr2=0x7fb23c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 -- 172.21.15.38:0/2528502401 shutdown_connections 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.082+0000 7fb2433bf640 1 -- 172.21.15.38:0/2528502401 wait complete. 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 Processor -- start 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 -- start start 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb23c1017a0 0x7fb23c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb23c103350 0x7fb23c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c10d420 con 0x7fb23c102990 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c10d2a0 con 0x7fb23c103350 2024-04-22T01:09:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb2433bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c10d5a0 con 0x7fb23c1017a0 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45092/0 (socket says 172.21.15.38:45092) 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 -- 172.21.15.38:0/95434759 learned_addr learned my addr 172.21.15.38:0/95434759 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24115b640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb23c1017a0 0x7fb23c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 -- 172.21.15.38:0/95434759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb23c1017a0 msgr2=0x7fb23c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb23c1017a0 0x7fb23c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 -- 172.21.15.38:0/95434759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb23c103350 msgr2=0x7fb23c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb23c103350 0x7fb23c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.083+0000 7fb24095a640 1 -- 172.21.15.38:0/95434759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c19f680 con 0x7fb23c102990 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.084+0000 7fb24095a640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c197fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009230 tx=0x7fb22c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.084+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c041070 con 0x7fb23c102990 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.084+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb22c0044d0 con 0x7fb23c102990 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.084+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23c19f910 con 0x7fb23c102990 2024-04-22T01:09:53.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.084+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c034440 con 0x7fb23c102990 2024-04-22T01:09:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.084+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23c19fe00 con 0x7fb23c102990 2024-04-22T01:09:53.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.085+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23c108a60 con 0x7fb23c102990 2024-04-22T01:09:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.086+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb22c044070 con 0x7fb23c102990 2024-04-22T01:09:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.086+0000 7fb2327fc640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2280a2b90 0x7fb2280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.086+0000 7fb24115b640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2280a2b90 0x7fb2280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.087+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb22c0f0820 con 0x7fb23c102990 2024-04-22T01:09:53.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.087+0000 7fb24115b640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2280a2b90 0x7fb2280a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb23c1027f0 tx=0x7fb238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:53.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.091+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb22c0bdc20 con 0x7fb23c102990 2024-04-22T01:09:53.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.274+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb23c02d3c0 con 0x7fb23c102990 2024-04-22T01:09:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.275+0000 7fb2327fc640 1 -- 172.21.15.38:0/95434759 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb22c0bd520 con 0x7fb23c102990 2024-04-22T01:09:53.275 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2280a2b90 msgr2=0x7fb2280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2280a2b90 0x7fb2280a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb23c1027f0 tx=0x7fb238005f90 comp rx=0 tx=0).stop 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 msgr2=0x7fb23c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c197fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009230 tx=0x7fb22c005d50 comp rx=0 tx=0).stop 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 shutdown_connections 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2280a2b90 0x7fb2280a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb23c103350 0x7fb23c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c102990 0x7fb23c197fa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 --2- 172.21.15.38:0/95434759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb23c1017a0 0x7fb23c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.276+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 >> 172.21.15.38:0/95434759 conn(0x7fb23c0fcd10 msgr2=0x7fb23c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.277+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 shutdown_connections 2024-04-22T01:09:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.277+0000 7fb2433bf640 1 -- 172.21.15.38:0/95434759 wait complete. 2024-04-22T01:09:53.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:09:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.389+0000 7f52208fb640 1 -- 172.21.15.38:0/321083394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c1006a0 msgr2=0x7f521c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 --2- 172.21.15.38:0/321083394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c1006a0 0x7f521c100ac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f520c009900 tx=0x7f520c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 -- 172.21.15.38:0/321083394 shutdown_connections 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 --2- 172.21.15.38:0/321083394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c102250 0x7f521c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 --2- 172.21.15.38:0/321083394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c101890 0x7f521c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 --2- 172.21.15.38:0/321083394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c1006a0 0x7f521c100ac0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 -- 172.21.15.38:0/321083394 >> 172.21.15.38:0/321083394 conn(0x7f521c0fbc30 msgr2=0x7f521c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 -- 172.21.15.38:0/321083394 shutdown_connections 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 -- 172.21.15.38:0/321083394 wait complete. 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 Processor -- start 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.390+0000 7f52208fb640 1 -- start start 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c1006a0 0x7f521c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c101890 0x7f521c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 0x7f521c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f521c10d440 con 0x7f521c102250 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f521c10d2c0 con 0x7f521c101890 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f521c10d5c0 con 0x7f521c1006a0 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 0x7f521c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 0x7f521c19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45120/0 (socket says 172.21.15.38:45120) 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 -- 172.21.15.38:0/1710192180 learned_addr learned my addr 172.21.15.38:0/1710192180 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521a59c640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c1006a0 0x7f521c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f5219d9b640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c101890 0x7f521c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 -- 172.21.15.38:0/1710192180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c1006a0 msgr2=0x7f521c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c1006a0 0x7f521c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 -- 172.21.15.38:0/1710192180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c101890 msgr2=0x7f521c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c101890 0x7f521c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 -- 172.21.15.38:0/1710192180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521c19f6f0 con 0x7f521c102250 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521a59c640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c1006a0 0x7f521c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f5219d9b640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c101890 0x7f521c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f521ad9d640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 0x7f521c19f080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f521400eb10 tx=0x7f521400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214004b20 con 0x7f521c102250 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f521c19f9e0 con 0x7f521c102250 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5214004cc0 con 0x7f521c102250 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.391+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214010760 con 0x7f521c102250 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.392+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f521c19fee0 con 0x7f521c102250 2024-04-22T01:09:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.393+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f521c107a40 con 0x7f521c102250 2024-04-22T01:09:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.394+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5214005020 con 0x7f521c102250 2024-04-22T01:09:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.394+0000 7f520b7fe640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f52040a2b90 0x7f52040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.394+0000 7f521a59c640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f52040a2b90 0x7f52040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.395+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f52140cf130 con 0x7f521c102250 2024-04-22T01:09:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.395+0000 7f521a59c640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f52040a2b90 0x7f52040a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f520c000c00 tx=0x7f520c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.399+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f521409c530 con 0x7f521c102250 2024-04-22T01:09:53.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.623+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f521c062340 con 0x7f521c102250 2024-04-22T01:09:53.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.623+0000 7f520b7fe640 1 -- 172.21.15.38:0/1710192180 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f521409be30 con 0x7f521c102250 2024-04-22T01:09:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:53.624 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-22T01:09:53.624 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 1 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f52040a2b90 msgr2=0x7f52040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f52040a2b90 0x7f52040a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f520c000c00 tx=0x7f520c005fd0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 msgr2=0x7f521c19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 0x7f521c19f080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f521400eb10 tx=0x7f521400c2d0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 shutdown_connections 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f52040a2b90 0x7f52040a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f521c102250 0x7f521c19f080 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f521c101890 0x7f521c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 --2- 172.21.15.38:0/1710192180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f521c1006a0 0x7f521c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:53.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.625+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 >> 172.21.15.38:0/1710192180 conn(0x7f521c0fbc30 msgr2=0x7f521c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:53.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.626+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 shutdown_connections 2024-04-22T01:09:53.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:53.626+0000 7f52208fb640 1 -- 172.21.15.38:0/1710192180 wait complete. 2024-04-22T01:09:53.738 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:53 smithi203 ceph-mon[31825]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:53.739 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1428828644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:09:53.739 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/95434759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:09:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:53 smithi111 ceph-mon[31057]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1428828644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:09:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/95434759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:09:53.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:53 smithi038 ceph-mon[28043]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:53.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1428828644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:09:53.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/95434759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:09:54.016 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:53 smithi203 systemd[1]: Starting Ceph mds.l for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:09:54.426 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 podman[62312]: 2024-04-22 01:09:54.037623954 +0000 UTC m=+0.018545203 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:09:54.426 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 podman[62312]: 2024-04-22 01:09:54.165784073 +0000 UTC m=+0.146705350 container create e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T01:09:54.426 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 podman[62312]: 2024-04-22 01:09:54.427121994 +0000 UTC m=+0.408043226 container init e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1710192180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6832/3645664141,v1:172.21.15.203:6833/3645664141] up:boot 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mon[31825]: fsmap 1 up:standby 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:54.766 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 podman[62312]: 2024-04-22 01:09:54.429774516 +0000 UTC m=+0.410695750 container start e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T01:09:54.767 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-mds[62331]: -- 172.21.15.203:0/1993643603 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563088ebba40 con 0x563088e8ec00 2024-04-22T01:09:54.767 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l[62327]: starting mds.l at 2024-04-22T01:09:54.767 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 bash[62312]: e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 2024-04-22T01:09:54.767 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:09:54 smithi203 systemd[1]: Started Ceph mds.l for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:09:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1710192180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:54 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6832/3645664141,v1:172.21.15.203:6833/3645664141] up:boot 2024-04-22T01:09:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:54 smithi111 ceph-mon[31057]: fsmap 1 up:standby 2024-04-22T01:09:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T01:09:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:54 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:54.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1710192180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:54 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6832/3645664141,v1:172.21.15.203:6833/3645664141] up:boot 2024-04-22T01:09:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:54 smithi038 ceph-mon[28043]: fsmap 1 up:standby 2024-04-22T01:09:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T01:09:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:54 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:55.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 -- 172.21.15.38:0/4057633609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144101870 msgr2=0x7f9144101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 --2- 172.21.15.38:0/4057633609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144101870 0x7f9144101cf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9130009900 tx=0x7f913002bc30 comp rx=0 tx=0).stop 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 -- 172.21.15.38:0/4057633609 shutdown_connections 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 --2- 172.21.15.38:0/4057633609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144102230 0x7f9144108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 --2- 172.21.15.38:0/4057633609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144101870 0x7f9144101cf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 --2- 172.21.15.38:0/4057633609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144100680 0x7f9144100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 -- 172.21.15.38:0/4057633609 >> 172.21.15.38:0/4057633609 conn(0x7f91440fbbf0 msgr2=0x7f91440fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 -- 172.21.15.38:0/4057633609 shutdown_connections 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.752+0000 7f914ade5640 1 -- 172.21.15.38:0/4057633609 wait complete. 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 Processor -- start 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 -- start start 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 0x7f914410ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144101870 0x7f91441094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144102230 0x7f9144109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f914410d420 con 0x7f9144100680 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f914410d2a0 con 0x7f9144102230 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f914ade5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f914410d5a0 con 0x7f9144101870 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f9148b81640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 0x7f914410ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f9148b81640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 0x7f914410ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45142/0 (socket says 172.21.15.38:45142) 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f9148b81640 1 -- 172.21.15.38:0/2552505992 learned_addr learned my addr 172.21.15.38:0/2552505992 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.753+0000 7f9149382640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144102230 0x7f9144109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9143fff640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144101870 0x7f91441094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9148b81640 1 -- 172.21.15.38:0/2552505992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144101870 msgr2=0x7f91441094c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9148b81640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144101870 0x7f91441094c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9148b81640 1 -- 172.21.15.38:0/2552505992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144102230 msgr2=0x7f9144109a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9148b81640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144102230 0x7f9144109a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9148b81640 1 -- 172.21.15.38:0/2552505992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914410a290 con 0x7f9144100680 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9143fff640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144101870 0x7f91441094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9149382640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144102230 0x7f9144109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9148b81640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 0x7f914410ae80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f913800e9c0 tx=0x7f913800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913800cd80 con 0x7f9144100680 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91380035f0 con 0x7f9144100680 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9138010690 con 0x7f9144100680 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91441a1bc0 con 0x7f9144100680 2024-04-22T01:09:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.754+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91441a20e0 con 0x7f9144100680 2024-04-22T01:09:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.755+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91440689b0 con 0x7f9144100680 2024-04-22T01:09:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.756+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f91380037a0 con 0x7f9144100680 2024-04-22T01:09:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.757+0000 7f9141ffb640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91340a2b90 0x7f91340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.757+0000 7f9143fff640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91340a2b90 0x7f91340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:55.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.757+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f91380cf130 con 0x7f9144100680 2024-04-22T01:09:55.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.758+0000 7f9143fff640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91340a2b90 0x7f91340a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9130006010 tx=0x7f913002c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:55.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.761+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f913809c490 con 0x7f9144100680 2024-04-22T01:09:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:55 smithi111 ceph-mon[31057]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:55 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:55 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:55 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:55 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:55.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:55 smithi111 ceph-mon[31057]: Deploying daemon mds.k on smithi111 2024-04-22T01:09:55.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:55 smithi038 ceph-mon[28043]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:55.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:55 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:55 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:55 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:55 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:55 smithi038 ceph-mon[28043]: Deploying daemon mds.k on smithi111 2024-04-22T01:09:55.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.984+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91441079a0 con 0x7f9144100680 2024-04-22T01:09:55.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.984+0000 7f9141ffb640 1 -- 172.21.15.38:0/2552505992 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f91441079a0 con 0x7f9144100680 2024-04-22T01:09:55.985 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:55.985 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T01:09:55.985 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 2 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91340a2b90 msgr2=0x7f91340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91340a2b90 0x7f91340a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9130006010 tx=0x7f913002c8a0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 msgr2=0x7f914410ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 0x7f914410ae80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f913800e9c0 tx=0x7f913800ee80 comp rx=0 tx=0).stop 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 shutdown_connections 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91340a2b90 0x7f91340a5050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9144102230 0x7f9144109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9144101870 0x7f91441094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 --2- 172.21.15.38:0/2552505992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9144100680 0x7f914410ae80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.986+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 >> 172.21.15.38:0/2552505992 conn(0x7f91440fbbf0 msgr2=0x7f9144104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.987+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 shutdown_connections 2024-04-22T01:09:55.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:55.987+0000 7f914ade5640 1 -- 172.21.15.38:0/2552505992 wait complete. 2024-04-22T01:09:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:55 smithi203 ceph-mon[31825]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:55 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:55 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:55 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:55 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:55 smithi203 ceph-mon[31825]: Deploying daemon mds.k on smithi111 2024-04-22T01:09:56.892 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2552505992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2552505992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:57.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2552505992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:57.145 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 podman[60660]: 2024-04-22 01:09:57.11671423 +0000 UTC m=+0.123287753 container create f82f4e5060e9093a43252a9bb32bbf936b7b408fe17030485dde11e22ca1ce32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T01:09:57.611 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 podman[60660]: 2024-04-22 01:09:57.336087777 +0000 UTC m=+0.342661301 container init f82f4e5060e9093a43252a9bb32bbf936b7b408fe17030485dde11e22ca1ce32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:09:57.611 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 podman[60660]: 2024-04-22 01:09:57.338646284 +0000 UTC m=+0.345219817 container start f82f4e5060e9093a43252a9bb32bbf936b7b408fe17030485dde11e22ca1ce32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8) 2024-04-22T01:09:57.611 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mds[60679]: -- 172.21.15.111:0/4095658637 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b26149fa40 con 0x55b261473000 2024-04-22T01:09:57.611 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k[60675]: starting mds.k at 2024-04-22T01:09:57.611 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 bash[60660]: f82f4e5060e9093a43252a9bb32bbf936b7b408fe17030485dde11e22ca1ce32 2024-04-22T01:09:57.611 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:09:57 smithi111 systemd[1]: Started Ceph mds.k for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:09:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:57.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:57.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:57 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:57.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:57.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:57 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:57.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:57.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:09:57.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:09:57.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:09:57.918 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:57 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:09:57.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:09:58.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 -- 172.21.15.38:0/1220430327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff4104b90 msgr2=0x7efff4106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 --2- 172.21.15.38:0/1220430327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff4104b90 0x7efff4106f80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7effe4009900 tx=0x7effe402bc30 comp rx=0 tx=0).stop 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 -- 172.21.15.38:0/1220430327 shutdown_connections 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 --2- 172.21.15.38:0/1220430327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff41074c0 0x7efff41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 --2- 172.21.15.38:0/1220430327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff4104b90 0x7efff4106f80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 --2- 172.21.15.38:0/1220430327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 0x7efff4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 -- 172.21.15.38:0/1220430327 >> 172.21.15.38:0/1220430327 conn(0x7efff40fbbf0 msgr2=0x7efff40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 -- 172.21.15.38:0/1220430327 shutdown_connections 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.110+0000 7efffa082640 1 -- 172.21.15.38:0/1220430327 wait complete. 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 Processor -- start 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 -- start start 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 0x7efff4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff4104b90 0x7efff4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 0x7efff419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:58.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efff410d450 con 0x7efff41074c0 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efff410d2d0 con 0x7efff4102260 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efffa082640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efff410d5d0 con 0x7efff4104b90 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 0x7efff419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 0x7efff419f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33620/0 (socket says 172.21.15.38:33620) 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 -- 172.21.15.38:0/1345517851 learned_addr learned my addr 172.21.15.38:0/1345517851 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff37fe640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 0x7efff4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff2ffd640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff4104b90 0x7efff4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 -- 172.21.15.38:0/1345517851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff4104b90 msgr2=0x7efff4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff4104b90 0x7efff4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 -- 172.21.15.38:0/1345517851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 msgr2=0x7efff4197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 0x7efff4197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff3fff640 1 -- 172.21.15.38:0/1345517851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff419f740 con 0x7efff41074c0 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff37fe640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 0x7efff4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:09:58.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.111+0000 7efff2ffd640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff4104b90 0x7efff4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:09:58.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.112+0000 7efff3fff640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 0x7efff419f0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7effec00b970 tx=0x7effec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:58.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.112+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec019070 con 0x7efff41074c0 2024-04-22T01:09:58.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.112+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effec00cb40 con 0x7efff41074c0 2024-04-22T01:09:58.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.112+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec007650 con 0x7efff41074c0 2024-04-22T01:09:58.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.112+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff419fa30 con 0x7efff41074c0 2024-04-22T01:09:58.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.112+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff419ff80 con 0x7efff41074c0 2024-04-22T01:09:58.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.113+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff40fd6e0 con 0x7efff41074c0 2024-04-22T01:09:58.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.114+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7effec01c070 con 0x7efff41074c0 2024-04-22T01:09:58.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.114+0000 7efff0ff9640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effdc0a2b90 0x7effdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:09:58.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.115+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7effec0ce1e0 con 0x7efff41074c0 2024-04-22T01:09:58.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.115+0000 7efff37fe640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effdc0a2b90 0x7effdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:09:58.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.115+0000 7efff37fe640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effdc0a2b90 0x7effdc0a5050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7effe8009b40 tx=0x7effe8005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:09:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.119+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7effec09b5e0 con 0x7efff41074c0 2024-04-22T01:09:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.346+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efff402d170 con 0x7efff41074c0 2024-04-22T01:09:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.347+0000 7efff0ff9640 1 -- 172.21.15.38:0/1345517851 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7effec09aee0 con 0x7efff41074c0 2024-04-22T01:09:58.347 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:09:58.347 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T01:09:58.347 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 3 2024-04-22T01:09:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effdc0a2b90 msgr2=0x7effdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effdc0a2b90 0x7effdc0a5050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7effe8009b40 tx=0x7effe8005e50 comp rx=0 tx=0).stop 2024-04-22T01:09:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 msgr2=0x7efff419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:09:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 0x7efff419f0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7effec00b970 tx=0x7effec00be30 comp rx=0 tx=0).stop 2024-04-22T01:09:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 shutdown_connections 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effdc0a2b90 0x7effdc0a5050 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff41074c0 0x7efff419f0d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff4104b90 0x7efff4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 --2- 172.21.15.38:0/1345517851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4102260 0x7efff4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 >> 172.21.15.38:0/1345517851 conn(0x7efff40fbbf0 msgr2=0x7efff4102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 shutdown_connections 2024-04-22T01:09:58.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:09:58.349+0000 7efffa082640 1 -- 172.21.15.38:0/1345517851 wait complete. 2024-04-22T01:09:58.884 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:58 smithi111 ceph-mon[31057]: Deploying daemon mds.j on smithi038 2024-04-22T01:09:58.884 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:58 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6832/1598672021,v1:172.21.15.111:6833/1598672021] up:boot 2024-04-22T01:09:58.884 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:58 smithi111 ceph-mon[31057]: fsmap 2 up:standby 2024-04-22T01:09:58.884 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:58 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T01:09:58.884 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:58 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1345517851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:58 smithi038 ceph-mon[28043]: Deploying daemon mds.j on smithi038 2024-04-22T01:09:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:58 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6832/1598672021,v1:172.21.15.111:6833/1598672021] up:boot 2024-04-22T01:09:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:58 smithi038 ceph-mon[28043]: fsmap 2 up:standby 2024-04-22T01:09:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:58 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T01:09:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:58 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1345517851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:58.928 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:58 smithi203 ceph-mon[31825]: Deploying daemon mds.j on smithi038 2024-04-22T01:09:58.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:58 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6832/1598672021,v1:172.21.15.111:6833/1598672021] up:boot 2024-04-22T01:09:58.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:58 smithi203 ceph-mon[31825]: fsmap 2 up:standby 2024-04-22T01:09:58.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:58 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T01:09:58.929 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:58 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1345517851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:09:59.884 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:09:59 smithi038 systemd[1]: Starting Ceph mds.j for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:09:59.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:09:59 smithi038 ceph-mon[28043]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:09:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:09:59 smithi111 ceph-mon[31057]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:00.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:09:59 smithi203 ceph-mon[31825]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:00.136 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 podman[67825]: 2024-04-22 01:10:00.010064665 +0000 UTC m=+0.018678421 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:00.136 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 podman[67825]: 2024-04-22 01:10:00.136835596 +0000 UTC m=+0.145449361 container create e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-22T01:10:00.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:00.386 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 podman[67825]: 2024-04-22 01:10:00.372831687 +0000 UTC m=+0.381445445 container init e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T01:10:00.386 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 podman[67825]: 2024-04-22 01:10:00.375639945 +0000 UTC m=+0.384253701 container start e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 -- 172.21.15.38:0/1733192940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c101890 msgr2=0x7ff75c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 --2- 172.21.15.38:0/1733192940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c101890 0x7ff75c101d10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009900 tx=0x7ff74c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 -- 172.21.15.38:0/1733192940 shutdown_connections 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 --2- 172.21.15.38:0/1733192940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c102250 0x7ff75c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 --2- 172.21.15.38:0/1733192940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c101890 0x7ff75c101d10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 --2- 172.21.15.38:0/1733192940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff75c1006a0 0x7ff75c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 -- 172.21.15.38:0/1733192940 >> 172.21.15.38:0/1733192940 conn(0x7ff75c0fbc30 msgr2=0x7ff75c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 -- 172.21.15.38:0/1733192940 shutdown_connections 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.470+0000 7ff7618fb640 1 -- 172.21.15.38:0/1733192940 wait complete. 2024-04-22T01:10:00.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 Processor -- start 2024-04-22T01:10:00.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 -- start start 2024-04-22T01:10:00.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff75c1006a0 0x7ff75c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:00.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c101890 0x7ff75c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:00.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 0x7ff75c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c10d440 con 0x7ff75c102250 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c10d2c0 con 0x7ff75c101890 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff7618fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c10d5c0 con 0x7ff75c1006a0 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 0x7ff75c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 0x7ff75c19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33646/0 (socket says 172.21.15.38:33646) 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 -- 172.21.15.38:0/2216176725 learned_addr learned my addr 172.21.15.38:0/2216176725 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75a7fc640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c101890 0x7ff75c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75affd640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff75c1006a0 0x7ff75c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 -- 172.21.15.38:0/2216176725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff75c1006a0 msgr2=0x7ff75c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff75c1006a0 0x7ff75c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 -- 172.21.15.38:0/2216176725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c101890 msgr2=0x7ff75c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c101890 0x7ff75c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75b7fe640 1 -- 172.21.15.38:0/2216176725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c19f710 con 0x7ff75c102250 2024-04-22T01:10:00.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.471+0000 7ff75a7fc640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c101890 0x7ff75c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.472+0000 7ff75b7fe640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 0x7ff75c19f0a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff75400eaa0 tx=0x7ff75400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.472+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75400cdb0 con 0x7ff75c102250 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.472+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff754005020 con 0x7ff75c102250 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.472+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75c19fa00 con 0x7ff75c102250 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.472+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7540116b0 con 0x7ff75c102250 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.472+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75c19ff20 con 0x7ff75c102250 2024-04-22T01:10:00.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.473+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75c1079e0 con 0x7ff75c102250 2024-04-22T01:10:00.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.474+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff754005560 con 0x7ff75c102250 2024-04-22T01:10:00.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.475+0000 7ff7608f9640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7480a2c10 0x7ff7480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:00.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.475+0000 7ff75affd640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7480a2c10 0x7ff7480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:00.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.475+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff7540ceb80 con 0x7ff75c102250 2024-04-22T01:10:00.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.475+0000 7ff75affd640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7480a2c10 0x7ff7480a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7440058c0 tx=0x7ff74400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:00.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.480+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff754016030 con 0x7ff75c102250 2024-04-22T01:10:00.644 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mds[67860]: -- 172.21.15.38:0/4075890460 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1c88bda40 con 0x55e1c8891000 2024-04-22T01:10:00.644 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j[67842]: starting mds.j at 2024-04-22T01:10:00.644 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 bash[67825]: e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 2024-04-22T01:10:00.644 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:10:00 smithi038 systemd[1]: Started Ceph mds.j for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:00.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.739+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff75c062340 con 0x7ff75c102250 2024-04-22T01:10:00.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.739+0000 7ff7608f9640 1 -- 172.21.15.38:0/2216176725 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7ff75409bee0 con 0x7ff75c102250 2024-04-22T01:10:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:00.740 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T01:10:00.740 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 4 2024-04-22T01:10:00.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7480a2c10 msgr2=0x7ff7480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:00.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7480a2c10 0x7ff7480a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7440058c0 tx=0x7ff74400a4c0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 msgr2=0x7ff75c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:00.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 0x7ff75c19f0a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff75400eaa0 tx=0x7ff75400ef60 comp rx=0 tx=0).stop 2024-04-22T01:10:00.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 shutdown_connections 2024-04-22T01:10:00.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7480a2c10 0x7ff7480a50d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff75c102250 0x7ff75c19f0a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff75c101890 0x7ff75c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 --2- 172.21.15.38:0/2216176725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff75c1006a0 0x7ff75c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:00.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 >> 172.21.15.38:0/2216176725 conn(0x7ff75c0fbc30 msgr2=0x7ff75c104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:00.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 shutdown_connections 2024-04-22T01:10:00.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:00.742+0000 7ff7618fb640 1 -- 172.21.15.38:0/2216176725 wait complete. 2024-04-22T01:10:00.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: overall HEALTH_OK 2024-04-22T01:10:00.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:00.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:00.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:00.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:00.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:00.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:00 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:00.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: overall HEALTH_OK 2024-04-22T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:00 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: overall HEALTH_OK 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:00 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:01 smithi038 ceph-mon[28043]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:01 smithi038 ceph-mon[28043]: Deploying daemon mds.d on smithi038 2024-04-22T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:01 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.38:6834/2658040728,v1:172.21.15.38:6835/2658040728] up:boot 2024-04-22T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:01 smithi038 ceph-mon[28043]: fsmap 3 up:standby 2024-04-22T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:01 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:01 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2216176725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:01 smithi203 ceph-mon[31825]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:01 smithi203 ceph-mon[31825]: Deploying daemon mds.d on smithi038 2024-04-22T01:10:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:01 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.38:6834/2658040728,v1:172.21.15.38:6835/2658040728] up:boot 2024-04-22T01:10:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:01 smithi203 ceph-mon[31825]: fsmap 3 up:standby 2024-04-22T01:10:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:01 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T01:10:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:01 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2216176725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:01 smithi111 ceph-mon[31057]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:01 smithi111 ceph-mon[31057]: Deploying daemon mds.d on smithi038 2024-04-22T01:10:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:01 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.38:6834/2658040728,v1:172.21.15.38:6835/2658040728] up:boot 2024-04-22T01:10:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:01 smithi111 ceph-mon[31057]: fsmap 3 up:standby 2024-04-22T01:10:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:01 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T01:10:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:01 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2216176725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:02.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:02.780 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:02 smithi038 systemd[1]: Starting Ceph mds.d for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 -- 172.21.15.38:0/10373494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0101890 msgr2=0x7f77c0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 --2- 172.21.15.38:0/10373494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0101890 0x7f77c0101d10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009900 tx=0x7f77b402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 -- 172.21.15.38:0/10373494 shutdown_connections 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 --2- 172.21.15.38:0/10373494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0102250 0x7f77c0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 --2- 172.21.15.38:0/10373494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0101890 0x7f77c0101d10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 --2- 172.21.15.38:0/10373494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c01006a0 0x7f77c0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 -- 172.21.15.38:0/10373494 >> 172.21.15.38:0/10373494 conn(0x7f77c00fbc30 msgr2=0x7f77c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 -- 172.21.15.38:0/10373494 shutdown_connections 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.869+0000 7f77c618a640 1 -- 172.21.15.38:0/10373494 wait complete. 2024-04-22T01:10:02.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 Processor -- start 2024-04-22T01:10:02.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 -- start start 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c01006a0 0x7f77c0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0101890 0x7f77c0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 0x7f77c019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77c010c300 con 0x7f77c0102250 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f77c010c180 con 0x7f77c0101890 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77c618a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77c010c480 con 0x7f77c01006a0 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 0x7f77c019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 0x7f77c019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33664/0 (socket says 172.21.15.38:33664) 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77bffff640 1 -- 172.21.15.38:0/3645748066 learned_addr learned my addr 172.21.15.38:0/3645748066 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.870+0000 7f77bf7fe640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c01006a0 0x7f77c0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77beffd640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0101890 0x7f77c0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bffff640 1 -- 172.21.15.38:0/3645748066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c01006a0 msgr2=0x7f77c0197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:02.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bffff640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c01006a0 0x7f77c0197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bffff640 1 -- 172.21.15.38:0/3645748066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0101890 msgr2=0x7f77c0198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bffff640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0101890 0x7f77c0198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bffff640 1 -- 172.21.15.38:0/3645748066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c019f700 con 0x7f77c0102250 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77beffd640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0101890 0x7f77c0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bffff640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 0x7f77c019f090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009fd0 tx=0x7f77b800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8017070 con 0x7f77c0102250 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8003e50 con 0x7f77c0102250 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c019f9f0 con 0x7f77c0102250 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b80106c0 con 0x7f77c0102250 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.871+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c019fef0 con 0x7f77c0102250 2024-04-22T01:10:02.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.872+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c01a0180 con 0x7f77c0102250 2024-04-22T01:10:02.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.873+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f77b8003970 con 0x7f77c0102250 2024-04-22T01:10:02.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.873+0000 7f77bcff9640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f77a80a2b90 0x7f77a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:02.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.874+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f77b80ce960 con 0x7f77c0102250 2024-04-22T01:10:02.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.874+0000 7f77bf7fe640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f77a80a2b90 0x7f77a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:02.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.874+0000 7f77bf7fe640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f77a80a2b90 0x7f77a80a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f77c01016f0 tx=0x7f77b000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:02.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:02.878+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f77b809bd60 con 0x7f77c0102250 2024-04-22T01:10:03.042 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:02 smithi038 podman[68196]: 2024-04-22 01:10:02.797370708 +0000 UTC m=+0.021541161 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:03.042 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:02 smithi038 podman[68196]: 2024-04-22 01:10:02.926147393 +0000 UTC m=+0.150317822 container create f1bf8db94a19bc683fc11e3b12aef7a10dee8c7b76903869ef4101fffd71ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T01:10:03.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.105+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77c01a0470 con 0x7f77c0102250 2024-04-22T01:10:03.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.106+0000 7f77bcff9640 1 -- 172.21.15.38:0/3645748066 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f77b809b660 con 0x7f77c0102250 2024-04-22T01:10:03.106 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:03.106 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T01:10:03.107 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 4 2024-04-22T01:10:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f77a80a2b90 msgr2=0x7f77a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f77a80a2b90 0x7f77a80a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f77c01016f0 tx=0x7f77b000b390 comp rx=0 tx=0).stop 2024-04-22T01:10:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 msgr2=0x7f77c019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 0x7f77c019f090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009fd0 tx=0x7f77b800b260 comp rx=0 tx=0).stop 2024-04-22T01:10:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 shutdown_connections 2024-04-22T01:10:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f77a80a2b90 0x7f77a80a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:03.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0102250 0x7f77c019f090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:03.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0101890 0x7f77c0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:03.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 --2- 172.21.15.38:0/3645748066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c01006a0 0x7f77c0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:03.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 >> 172.21.15.38:0/3645748066 conn(0x7f77c00fbc30 msgr2=0x7f77c0069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:03.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 shutdown_connections 2024-04-22T01:10:03.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:03.108+0000 7f77c618a640 1 -- 172.21.15.38:0/3645748066 wait complete. 2024-04-22T01:10:03.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:03 smithi038 podman[68196]: 2024-04-22 01:10:03.154161086 +0000 UTC m=+0.378331516 container init f1bf8db94a19bc683fc11e3b12aef7a10dee8c7b76903869ef4101fffd71ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5) 2024-04-22T01:10:03.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:03 smithi038 podman[68196]: 2024-04-22 01:10:03.156786063 +0000 UTC m=+0.380956493 container start f1bf8db94a19bc683fc11e3b12aef7a10dee8c7b76903869ef4101fffd71ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T01:10:03.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mds[68242]: -- 172.21.15.38:0/2660273680 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563bf1585a40 con 0x563bf1558800 2024-04-22T01:10:03.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d[68236]: starting mds.d at 2024-04-22T01:10:03.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:03 smithi038 bash[68196]: f1bf8db94a19bc683fc11e3b12aef7a10dee8c7b76903869ef4101fffd71ff99 2024-04-22T01:10:03.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:10:03 smithi038 systemd[1]: Started Ceph mds.d for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:03.978 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:03.978 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3645748066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:03.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:03.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:03.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:03.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:03.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:03.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:03 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3645748066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:04.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:03 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3645748066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:04.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:03 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:04 smithi203 ceph-mon[31825]: Deploying daemon mds.e on smithi111 2024-04-22T01:10:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:04 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.38:6836/3419129141,v1:172.21.15.38:6837/3419129141] up:boot 2024-04-22T01:10:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:04 smithi203 ceph-mon[31825]: fsmap 4 up:standby 2024-04-22T01:10:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:04 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T01:10:05.022 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:04 smithi111 ceph-mon[31057]: Deploying daemon mds.e on smithi111 2024-04-22T01:10:05.023 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:04 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.38:6836/3419129141,v1:172.21.15.38:6837/3419129141] up:boot 2024-04-22T01:10:05.023 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:04 smithi111 ceph-mon[31057]: fsmap 4 up:standby 2024-04-22T01:10:05.023 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:04 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T01:10:05.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:05.147 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:04 smithi038 ceph-mon[28043]: Deploying daemon mds.e on smithi111 2024-04-22T01:10:05.147 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:04 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.38:6836/3419129141,v1:172.21.15.38:6837/3419129141] up:boot 2024-04-22T01:10:05.147 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:04 smithi038 ceph-mon[28043]: fsmap 4 up:standby 2024-04-22T01:10:05.147 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:04 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 -- 172.21.15.38:0/192885394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec100680 msgr2=0x7efdec100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/192885394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec100680 0x7efdec100aa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efde0009900 tx=0x7efde002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 -- 172.21.15.38:0/192885394 shutdown_connections 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/192885394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdec102230 0x7efdec108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/192885394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 0x7efdec101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/192885394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec100680 0x7efdec100aa0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 -- 172.21.15.38:0/192885394 >> 172.21.15.38:0/192885394 conn(0x7efdec0fbbf0 msgr2=0x7efdec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.235+0000 7efdf2f8a640 1 -- 172.21.15.38:0/192885394 shutdown_connections 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 -- 172.21.15.38:0/192885394 wait complete. 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 Processor -- start 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 -- start start 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdec100680 0x7efdec0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 0x7efdec19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 0x7efdec19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efdec10c340 con 0x7efdec102230 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdec10c1c0 con 0x7efdec100680 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf2f8a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efdec10c4c0 con 0x7efdec101870 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf1527640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 0x7efdec19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf1527640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 0x7efdec19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33706/0 (socket says 172.21.15.38:33706) 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.236+0000 7efdf1527640 1 -- 172.21.15.38:0/3225647830 learned_addr learned my addr 172.21.15.38:0/3225647830 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf0d26640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdec100680 0x7efdec0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdebfff640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 0x7efdec19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf1527640 1 -- 172.21.15.38:0/3225647830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 msgr2=0x7efdec19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf1527640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 0x7efdec19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf1527640 1 -- 172.21.15.38:0/3225647830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdec100680 msgr2=0x7efdec0694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf1527640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdec100680 0x7efdec0694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf1527640 1 -- 172.21.15.38:0/3225647830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec19f6d0 con 0x7efdec102230 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdebfff640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 0x7efdec19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf1527640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 0x7efdec19ee10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efde400eaa0 tx=0x7efde400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde400cdb0 con 0x7efdec102230 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efde4005020 con 0x7efdec102230 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdec19f9c0 con 0x7efdec102230 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde40116b0 con 0x7efdec102230 2024-04-22T01:10:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.237+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdec19fec0 con 0x7efdec102230 2024-04-22T01:10:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.238+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdec1079a0 con 0x7efdec102230 2024-04-22T01:10:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.239+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efde4005560 con 0x7efdec102230 2024-04-22T01:10:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.240+0000 7efde9ffb640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efddc0a2b90 0x7efddc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.240+0000 7efdf0d26640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efddc0a2b90 0x7efddc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.240+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efde40cee90 con 0x7efdec102230 2024-04-22T01:10:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.240+0000 7efdf0d26640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efddc0a2b90 0x7efddc0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efde002c170 tx=0x7efde002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:05.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.244+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efde4016030 con 0x7efdec102230 2024-04-22T01:10:05.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.469+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdec02d170 con 0x7efdec102230 2024-04-22T01:10:05.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.470+0000 7efde9ffb640 1 -- 172.21.15.38:0/3225647830 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7efde40ce020 con 0x7efdec102230 2024-04-22T01:10:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:05.471 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T01:10:05.471 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 5 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efddc0a2b90 msgr2=0x7efddc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efddc0a2b90 0x7efddc0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efde002c170 tx=0x7efde002df90 comp rx=0 tx=0).stop 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 msgr2=0x7efdec19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 0x7efdec19ee10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efde400eaa0 tx=0x7efde400ef60 comp rx=0 tx=0).stop 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 shutdown_connections 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efddc0a2b90 0x7efddc0a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdec102230 0x7efdec19ee10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdec101870 0x7efdec19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 --2- 172.21.15.38:0/3225647830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdec100680 0x7efdec0694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 >> 172.21.15.38:0/3225647830 conn(0x7efdec0fbbf0 msgr2=0x7efdec104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 shutdown_connections 2024-04-22T01:10:05.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:05.472+0000 7efdf2f8a640 1 -- 172.21.15.38:0/3225647830 wait complete. 2024-04-22T01:10:05.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:05 smithi111 ceph-mon[31057]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:05.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:05 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3225647830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:05 smithi038 ceph-mon[28043]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3225647830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:05 smithi203 ceph-mon[31825]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:06.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:05 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3225647830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:06.395 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:10:06 smithi111 systemd[1]: Started Ceph mds.e for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:07.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:07.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:07.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:07.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: Deploying daemon mds.f on smithi203 2024-04-22T01:10:07.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] up:boot 2024-04-22T01:10:07.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: fsmap 5 up:standby 2024-04-22T01:10:07.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T01:10:07.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 -- 172.21.15.38:0/1782030536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4104b30 msgr2=0x7f15a4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:07.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 --2- 172.21.15.38:0/1782030536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4104b30 0x7f15a4106f20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1594009900 tx=0x7f159402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 -- 172.21.15.38:0/1782030536 shutdown_connections 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 --2- 172.21.15.38:0/1782030536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4107460 0x7f15a41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 --2- 172.21.15.38:0/1782030536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4104b30 0x7f15a4106f20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 --2- 172.21.15.38:0/1782030536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15a4102200 0x7f15a41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 -- 172.21.15.38:0/1782030536 >> 172.21.15.38:0/1782030536 conn(0x7f15a40fbbf0 msgr2=0x7f15a40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 -- 172.21.15.38:0/1782030536 shutdown_connections 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 -- 172.21.15.38:0/1782030536 wait complete. 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 Processor -- start 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.600+0000 7f15ab615640 1 -- start start 2024-04-22T01:10:07.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 0x7f15a4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15a4104b30 0x7f15a4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4107460 0x7f15a419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15a410d440 con 0x7f15a4102200 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15a410d2c0 con 0x7f15a4107460 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15a410d5c0 con 0x7f15a4104b30 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 0x7f15a4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 0x7f15a4197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33202/0 (socket says 172.21.15.38:33202) 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 -- 172.21.15.38:0/104908437 learned_addr learned my addr 172.21.15.38:0/104908437 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a9bb2640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4107460 0x7f15a419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a8bb0640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15a4104b30 0x7f15a4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 -- 172.21.15.38:0/104908437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15a4104b30 msgr2=0x7f15a4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15a4104b30 0x7f15a4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 -- 172.21.15.38:0/104908437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4107460 msgr2=0x7f15a419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:07.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4107460 0x7f15a419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 -- 172.21.15.38:0/104908437 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a419f6d0 con 0x7f15a4102200 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15a93b1640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 0x7f15a4197ab0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15a000d950 tx=0x7f15a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0017030 con 0x7f15a4102200 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a0004020 con 0x7f15a4102200 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a419f9c0 con 0x7f15a4102200 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.601+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a00047a0 con 0x7f15a4102200 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.602+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a419ff30 con 0x7f15a4102200 2024-04-22T01:10:07.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.602+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a40fd6e0 con 0x7f15a4102200 2024-04-22T01:10:07.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.604+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f15a00041d0 con 0x7f15a4102200 2024-04-22T01:10:07.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.604+0000 7f159a7fc640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15900a2c10 0x7f15900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:07.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.604+0000 7f15a8bb0640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15900a2c10 0x7f15900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:07.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.604+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15a00ce590 con 0x7f15a4102200 2024-04-22T01:10:07.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.605+0000 7f15a8bb0640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15900a2c10 0x7f15900a50d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f159402c170 tx=0x7f159402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:07.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.609+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f15a009b990 con 0x7f15a4102200 2024-04-22T01:10:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: Deploying daemon mds.f on smithi203 2024-04-22T01:10:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] up:boot 2024-04-22T01:10:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: fsmap 5 up:standby 2024-04-22T01:10:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: Deploying daemon mds.f on smithi203 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] up:boot 2024-04-22T01:10:07.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: fsmap 5 up:standby 2024-04-22T01:10:07.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T01:10:07.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.832+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15a402d170 con 0x7f15a4102200 2024-04-22T01:10:07.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.832+0000 7f159a7fc640 1 -- 172.21.15.38:0/104908437 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f15a009b290 con 0x7f15a4102200 2024-04-22T01:10:07.833 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:07.833 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T01:10:07.833 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 6 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15900a2c10 msgr2=0x7f15900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15900a2c10 0x7f15900a50d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f159402c170 tx=0x7f159402df90 comp rx=0 tx=0).stop 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 msgr2=0x7f15a4197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 0x7f15a4197ab0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15a000d950 tx=0x7f15a000de10 comp rx=0 tx=0).stop 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 shutdown_connections 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15900a2c10 0x7f15900a50d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4107460 0x7f15a419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15a4104b30 0x7f15a4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 --2- 172.21.15.38:0/104908437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15a4102200 0x7f15a4197ab0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:07.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 >> 172.21.15.38:0/104908437 conn(0x7f15a40fbbf0 msgr2=0x7f15a4102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:07.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 shutdown_connections 2024-04-22T01:10:07.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:07.835+0000 7f15ab615640 1 -- 172.21.15.38:0/104908437 wait complete. 2024-04-22T01:10:08.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:08 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/104908437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:08.516 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 systemd[1]: Starting Ceph mds.f for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:10:08.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:08 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/104908437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:08.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:08 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/104908437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:08.907 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 podman[62618]: 2024-04-22 01:10:08.529606823 +0000 UTC m=+0.018750972 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:08.907 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 podman[62618]: 2024-04-22 01:10:08.646692297 +0000 UTC m=+0.135836435 container create aaf036dce494b6a7919c96b5f2d0ef21685284ce9c2375cd4c5546e299a912a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-22T01:10:08.907 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 podman[62618]: 2024-04-22 01:10:08.908231477 +0000 UTC m=+0.397375615 container init aaf036dce494b6a7919c96b5f2d0ef21685284ce9c2375cd4c5546e299a912a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T01:10:09.266 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 podman[62618]: 2024-04-22 01:10:08.910857514 +0000 UTC m=+0.400001663 container start aaf036dce494b6a7919c96b5f2d0ef21685284ce9c2375cd4c5546e299a912a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-22T01:10:09.266 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 ceph-mds[62637]: -- 172.21.15.203:0/3448973324 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e5efa6fa40 con 0x55e5efa43000 2024-04-22T01:10:09.267 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f[62633]: starting mds.f at 2024-04-22T01:10:09.267 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:09 smithi203 bash[62618]: aaf036dce494b6a7919c96b5f2d0ef21685284ce9c2375cd4c5546e299a912a6 2024-04-22T01:10:09.267 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:10:09 smithi203 systemd[1]: Started Ceph mds.f for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:09.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:09.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:09.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:09.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:09 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:09.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:09 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:09.722 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:09 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:09.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:09.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.962+0000 7f8d5340e640 1 -- 172.21.15.38:0/3495453943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c102200 msgr2=0x7f8d4c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:09.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.962+0000 7f8d5340e640 1 --2- 172.21.15.38:0/3495453943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c102200 0x7f8d4c1045f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8d48009900 tx=0x7f8d4802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:09.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.962+0000 7f8d5340e640 1 -- 172.21.15.38:0/3495453943 shutdown_connections 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.962+0000 7f8d5340e640 1 --2- 172.21.15.38:0/3495453943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c107460 0x7f8d4c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 --2- 172.21.15.38:0/3495453943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d4c104b30 0x7f8d4c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 --2- 172.21.15.38:0/3495453943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c102200 0x7f8d4c1045f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- 172.21.15.38:0/3495453943 >> 172.21.15.38:0/3495453943 conn(0x7f8d4c0fbbf0 msgr2=0x7f8d4c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- 172.21.15.38:0/3495453943 shutdown_connections 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- 172.21.15.38:0/3495453943 wait complete. 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 Processor -- start 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- start start 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c102200 0x7f8d4c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 0x7f8d4c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d4c107460 0x7f8d4c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c10d450 con 0x7f8d4c104b30 2024-04-22T01:10:09.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c10d2d0 con 0x7f8d4c102200 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.963+0000 7f8d5340e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c10d5d0 con 0x7f8d4c107460 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 0x7f8d4c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 0x7f8d4c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33232/0 (socket says 172.21.15.38:33232) 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 -- 172.21.15.38:0/2759049153 learned_addr learned my addr 172.21.15.38:0/2759049153 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d511aa640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c102200 0x7f8d4c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d519ab640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d4c107460 0x7f8d4c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 -- 172.21.15.38:0/2759049153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d4c107460 msgr2=0x7f8d4c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d4c107460 0x7f8d4c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 -- 172.21.15.38:0/2759049153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c102200 msgr2=0x7f8d4c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c102200 0x7f8d4c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 -- 172.21.15.38:0/2759049153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4c19f6e0 con 0x7f8d4c104b30 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d511aa640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c102200 0x7f8d4c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d509a9640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 0x7f8d4c198000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c005fd0 tx=0x7f8d3c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c017070 con 0x7f8d4c104b30 2024-04-22T01:10:09.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d4c19f9d0 con 0x7f8d4c104b30 2024-04-22T01:10:09.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3c009c30 con 0x7f8d4c104b30 2024-04-22T01:10:09.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c010460 con 0x7f8d4c104b30 2024-04-22T01:10:09.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.964+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d4c19ff20 con 0x7f8d4c104b30 2024-04-22T01:10:09.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.965+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d4c0fd6e0 con 0x7f8d4c104b30 2024-04-22T01:10:09.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.967+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8d3c003e70 con 0x7f8d4c104b30 2024-04-22T01:10:09.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.968+0000 7f8d427fc640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d380a2b90 0x7f8d380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:09.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.968+0000 7f8d511aa640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d380a2b90 0x7f8d380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:09.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.968+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8d3c0ce820 con 0x7f8d4c104b30 2024-04-22T01:10:09.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.969+0000 7f8d511aa640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d380a2b90 0x7f8d380a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8d4802c240 tx=0x7f8d4802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:09.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:09.973+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8d3c020030 con 0x7f8d4c104b30 2024-04-22T01:10:10.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.198+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d4c02d170 con 0x7f8d4c104b30 2024-04-22T01:10:10.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.198+0000 7f8d427fc640 1 -- 172.21.15.38:0/2759049153 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f8d3c09bc20 con 0x7f8d4c104b30 2024-04-22T01:10:10.199 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:10.199 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T01:10:10.199 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 7 2024-04-22T01:10:10.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.200+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d380a2b90 msgr2=0x7f8d380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:10.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.200+0000 7f8d5340e640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d380a2b90 0x7f8d380a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8d4802c240 tx=0x7f8d4802df90 comp rx=0 tx=0).stop 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.200+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 msgr2=0x7f8d4c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.200+0000 7f8d5340e640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 0x7f8d4c198000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c005fd0 tx=0x7f8d3c00dde0 comp rx=0 tx=0).stop 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 shutdown_connections 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d380a2b90 0x7f8d380a5050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d4c107460 0x7f8d4c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d4c104b30 0x7f8d4c198000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 --2- 172.21.15.38:0/2759049153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4c102200 0x7f8d4c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 >> 172.21.15.38:0/2759049153 conn(0x7f8d4c0fbbf0 msgr2=0x7f8d4c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 shutdown_connections 2024-04-22T01:10:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:10.201+0000 7f8d5340e640 1 -- 172.21.15.38:0/2759049153 wait complete. 2024-04-22T01:10:10.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:10 smithi038 ceph-mon[28043]: Deploying daemon mds.c on smithi203 2024-04-22T01:10:10.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:10 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6834/4027190739,v1:172.21.15.203:6835/4027190739] up:boot 2024-04-22T01:10:10.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:10 smithi038 ceph-mon[28043]: fsmap 6 up:standby 2024-04-22T01:10:10.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T01:10:10.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:10 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2759049153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:10.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:10 smithi203 ceph-mon[31825]: Deploying daemon mds.c on smithi203 2024-04-22T01:10:10.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:10 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6834/4027190739,v1:172.21.15.203:6835/4027190739] up:boot 2024-04-22T01:10:10.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:10 smithi203 ceph-mon[31825]: fsmap 6 up:standby 2024-04-22T01:10:10.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T01:10:10.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:10 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2759049153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:10.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:10 smithi111 ceph-mon[31057]: Deploying daemon mds.c on smithi203 2024-04-22T01:10:10.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:10 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6834/4027190739,v1:172.21.15.203:6835/4027190739] up:boot 2024-04-22T01:10:10.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:10 smithi111 ceph-mon[31057]: fsmap 6 up:standby 2024-04-22T01:10:10.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T01:10:10.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:10 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2759049153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:11.292 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 systemd[1]: Starting Ceph mds.c for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:10:11.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:11 smithi038 ceph-mon[28043]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:11.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:11 smithi203 ceph-mon[31825]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:11.593 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 podman[62925]: 2024-04-22 01:10:11.345113596 +0000 UTC m=+0.018680430 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:11.593 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 podman[62925]: 2024-04-22 01:10:11.471450939 +0000 UTC m=+0.145017745 container create a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:10:11.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:11 smithi111 ceph-mon[31057]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:12.016 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 podman[62925]: 2024-04-22 01:10:11.743370643 +0000 UTC m=+0.416937454 container init a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-22T01:10:12.016 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 podman[62925]: 2024-04-22 01:10:11.745899796 +0000 UTC m=+0.419466612 container start a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-820a2f8) 2024-04-22T01:10:12.016 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 ceph-mds[62943]: -- 172.21.15.203:0/252584526 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55915b8f1a40 con 0x55915b8c4c00 2024-04-22T01:10:12.016 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c[62939]: starting mds.c at 2024-04-22T01:10:12.016 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 bash[62925]: a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 2024-04-22T01:10:12.016 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:10:11 smithi203 systemd[1]: Started Ceph mds.c for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:12.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:12.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4144717281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce01006a0 msgr2=0x7f9ce0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:12.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4144717281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce01006a0 0x7f9ce0100ac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009900 tx=0x7f9cd402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4144717281 shutdown_connections 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4144717281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce0102250 0x7f9ce0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4144717281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 0x7f9ce0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4144717281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce01006a0 0x7f9ce0100ac0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4144717281 >> 172.21.15.38:0/4144717281 conn(0x7f9ce00fbc30 msgr2=0x7f9ce00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4144717281 shutdown_connections 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.332+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4144717281 wait complete. 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 Processor -- start 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 -- start start 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01006a0 0x7f9ce0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 0x7f9ce019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:12.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 0x7f9ce01a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce010d440 con 0x7f9ce0102250 2024-04-22T01:10:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce010d2c0 con 0x7f9ce0101890 2024-04-22T01:10:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce6f85640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce010d5c0 con 0x7f9ce01006a0 2024-04-22T01:10:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce5522640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 0x7f9ce01a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9cdffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 0x7f9ce019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce5522640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 0x7f9ce01a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33248/0 (socket says 172.21.15.38:33248) 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce5522640 1 -- 172.21.15.38:0/4163654167 learned_addr learned my addr 172.21.15.38:0/4163654167 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.333+0000 7f9ce4d21640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01006a0 0x7f9ce0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cdffff640 1 -- 172.21.15.38:0/4163654167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01006a0 msgr2=0x7f9ce0199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cdffff640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01006a0 0x7f9ce0199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cdffff640 1 -- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 msgr2=0x7f9ce01a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cdffff640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 0x7f9ce01a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cdffff640 1 -- 172.21.15.38:0/4163654167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce01a1910 con 0x7f9ce0101890 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9ce4d21640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01006a0 0x7f9ce0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9ce5522640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 0x7f9ce01a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cdffff640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 0x7f9ce019a2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e9f0 tx=0x7f9ccc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00cd80 con 0x7f9ce0101890 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce01a1c00 con 0x7f9ce0101890 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ccc0035f0 con 0x7f9ce0101890 2024-04-22T01:10:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc010670 con 0x7f9ce0101890 2024-04-22T01:10:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.334+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce01a2150 con 0x7f9ce0101890 2024-04-22T01:10:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.335+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce01a2380 con 0x7f9ce0101890 2024-04-22T01:10:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.336+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9ccc0037a0 con 0x7f9ce0101890 2024-04-22T01:10:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.337+0000 7f9cddffb640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9cd00a2b90 0x7f9cd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.337+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9ccc0cefc0 con 0x7f9ce0101890 2024-04-22T01:10:12.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.337+0000 7f9ce4d21640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9cd00a2b90 0x7f9cd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:12.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.337+0000 7f9ce4d21640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9cd00a2b90 0x7f9cd00a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ce019b1d0 tx=0x7f9cd4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:12.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.341+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9ccc09c3c0 con 0x7f9ce0101890 2024-04-22T01:10:12.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.565+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ce01a2550 con 0x7f9ce0101890 2024-04-22T01:10:12.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.567+0000 7f9cddffb640 1 -- 172.21.15.38:0/4163654167 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7f9ccc09bcc0 con 0x7f9ce0101890 2024-04-22T01:10:12.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T01:10:12.568 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 8 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9cd00a2b90 msgr2=0x7f9cd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9cd00a2b90 0x7f9cd00a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ce019b1d0 tx=0x7f9cd4005fd0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 msgr2=0x7f9ce019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 0x7f9ce019a2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e9f0 tx=0x7f9ccc00eeb0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 shutdown_connections 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9cd00a2b90 0x7f9cd00a5050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ce0102250 0x7f9ce01a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ce0101890 0x7f9ce019a2e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 --2- 172.21.15.38:0/4163654167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01006a0 0x7f9ce0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 >> 172.21.15.38:0/4163654167 conn(0x7f9ce00fbc30 msgr2=0x7f9ce01094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:12.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 shutdown_connections 2024-04-22T01:10:12.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:12.568+0000 7f9ce6f85640 1 -- 172.21.15.38:0/4163654167 wait complete. 2024-04-22T01:10:13.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:13.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: Deploying daemon mds.h on smithi111 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] up:boot 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: fsmap 7 up:standby 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T01:10:13.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:12 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4163654167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: Deploying daemon mds.h on smithi111 2024-04-22T01:10:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] up:boot 2024-04-22T01:10:13.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: fsmap 7 up:standby 2024-04-22T01:10:13.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T01:10:13.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:12 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4163654167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:13.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:13.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: Deploying daemon mds.h on smithi111 2024-04-22T01:10:13.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] up:boot 2024-04-22T01:10:13.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: fsmap 7 up:standby 2024-04-22T01:10:13.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T01:10:13.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:12 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4163654167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:14.395 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 systemd[1]: Starting Ceph mds.h for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:10:14.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.694+0000 7fe96a6c6640 1 -- 172.21.15.38:0/283338682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9641006a0 msgr2=0x7fe964100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.694+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/283338682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9641006a0 0x7fe964100ac0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe954009900 tx=0x7fe95402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.694+0000 7fe96a6c6640 1 -- 172.21.15.38:0/283338682 shutdown_connections 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.694+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/283338682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe964102250 0x7fe964108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.694+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/283338682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 0x7fe964101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.694+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/283338682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9641006a0 0x7fe964100ac0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 -- 172.21.15.38:0/283338682 >> 172.21.15.38:0/283338682 conn(0x7fe9640fbc30 msgr2=0x7fe9640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 -- 172.21.15.38:0/283338682 shutdown_connections 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 -- 172.21.15.38:0/283338682 wait complete. 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 Processor -- start 2024-04-22T01:10:14.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 -- start start 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9641006a0 0x7fe96406a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.695+0000 7fe96a6c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 0x7fe96419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe96a6c6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 0x7fe96419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe96a6c6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe96410c2f0 con 0x7fe964102250 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe96a6c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe96410c170 con 0x7fe964101890 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe96a6c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe96410c470 con 0x7fe9641006a0 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe968c63640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 0x7fe96419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 0x7fe96419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe968c63640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 0x7fe96419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33264/0 (socket says 172.21.15.38:33264) 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe968c63640 1 -- 172.21.15.38:0/3321884142 learned_addr learned my addr 172.21.15.38:0/3321884142 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe963fff640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9641006a0 0x7fe96406a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:14.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 -- 172.21.15.38:0/3321884142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9641006a0 msgr2=0x7fe96406a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9641006a0 0x7fe96406a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 -- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 msgr2=0x7fe96419ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 0x7fe96419ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 -- 172.21.15.38:0/3321884142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96419f710 con 0x7fe964101890 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe968c63640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 0x7fe96419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9637fe640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 0x7fe96419c7e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe94c005fd0 tx=0x7fe94c00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c009c30 con 0x7fe964101890 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe94c009dd0 con 0x7fe964101890 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe96419fa00 con 0x7fe964101890 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.696+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c010460 con 0x7fe964101890 2024-04-22T01:10:14.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.697+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96419ffa0 con 0x7fe964101890 2024-04-22T01:10:14.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.697+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe964107a40 con 0x7fe964101890 2024-04-22T01:10:14.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.699+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe94c003ef0 con 0x7fe964101890 2024-04-22T01:10:14.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.699+0000 7fe9617fa640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9500a2c10 0x7fe9500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:14.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.699+0000 7fe963fff640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9500a2c10 0x7fe9500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:14.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.699+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe94c0ce6d0 con 0x7fe964101890 2024-04-22T01:10:14.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.700+0000 7fe963fff640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9500a2c10 0x7fe9500a50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe95402c7b0 tx=0x7fe954005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:14.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.703+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe94c017020 con 0x7fe964101890 2024-04-22T01:10:14.759 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 podman[61285]: 2024-04-22 01:10:14.36553975 +0000 UTC m=+0.018021837 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:14.759 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 podman[61285]: 2024-04-22 01:10:14.489560513 +0000 UTC m=+0.142042568 container create a7209b95fa1783210cf1167f2040a52a74b16d255b803319e84a579bde2ce680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T01:10:14.759 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 podman[61285]: 2024-04-22 01:10:14.759186793 +0000 UTC m=+0.411668847 container init a7209b95fa1783210cf1167f2040a52a74b16d255b803319e84a579bde2ce680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-22T01:10:14.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.927+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe96402d3c0 con 0x7fe964101890 2024-04-22T01:10:14.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.930+0000 7fe9617fa640 1 -- 172.21.15.38:0/3321884142 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fe94c09bb50 con 0x7fe964101890 2024-04-22T01:10:14.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:14.930 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T01:10:14.930 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 9 2024-04-22T01:10:14.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9500a2c10 msgr2=0x7fe9500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:14.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9500a2c10 0x7fe9500a50d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe95402c7b0 tx=0x7fe954005fd0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 msgr2=0x7fe96419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:14.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 0x7fe96419c7e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe94c005fd0 tx=0x7fe94c00dde0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 shutdown_connections 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9500a2c10 0x7fe9500a50d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe964102250 0x7fe96419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe964101890 0x7fe96419c7e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 --2- 172.21.15.38:0/3321884142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9641006a0 0x7fe96406a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 >> 172.21.15.38:0/3321884142 conn(0x7fe9640fbc30 msgr2=0x7fe964104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 shutdown_connections 2024-04-22T01:10:14.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:14.932+0000 7fe96a6c6640 1 -- 172.21.15.38:0/3321884142 wait complete. 2024-04-22T01:10:15.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:14 smithi111 ceph-mon[31057]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:15.145 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 podman[61285]: 2024-04-22 01:10:14.762119622 +0000 UTC m=+0.414601685 container start a7209b95fa1783210cf1167f2040a52a74b16d255b803319e84a579bde2ce680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:10:15.146 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 ceph-mds[61303]: -- 172.21.15.111:0/2760613036 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56488c51da40 con 0x56488c4f0c00 2024-04-22T01:10:15.146 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h[61299]: starting mds.h at 2024-04-22T01:10:15.146 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 bash[61285]: a7209b95fa1783210cf1167f2040a52a74b16d255b803319e84a579bde2ce680 2024-04-22T01:10:15.146 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:10:14 smithi111 systemd[1]: Started Ceph mds.h for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:15.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:14 smithi203 ceph-mon[31825]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:15.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:14 smithi038 ceph-mon[28043]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3321884142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] up:boot 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: fsmap 8 up:standby 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:16.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:16.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:16.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:15 smithi203 ceph-mon[31825]: Deploying daemon mds.i on smithi203 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3321884142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] up:boot 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: fsmap 8 up:standby 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:16.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:16.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:16.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:15 smithi111 ceph-mon[31057]: Deploying daemon mds.i on smithi203 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3321884142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] up:boot 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: fsmap 8 up:standby 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:15 smithi038 ceph-mon[28043]: Deploying daemon mds.i on smithi203 2024-04-22T01:10:16.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:17.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 -- 172.21.15.38:0/1647703568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 msgr2=0x7fa664101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 --2- 172.21.15.38:0/1647703568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664101bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa660009900 tx=0x7fa66002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 -- 172.21.15.38:0/1647703568 shutdown_connections 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 --2- 172.21.15.38:0/1647703568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa664103360 0x7fa664109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 --2- 172.21.15.38:0/1647703568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6641029a0 0x7fa664102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 --2- 172.21.15.38:0/1647703568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664101bd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 -- 172.21.15.38:0/1647703568 >> 172.21.15.38:0/1647703568 conn(0x7fa6640fcd40 msgr2=0x7fa6640ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 -- 172.21.15.38:0/1647703568 shutdown_connections 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.060+0000 7fa66b75b640 1 -- 172.21.15.38:0/1647703568 wait complete. 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 Processor -- start 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 -- start start 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:17.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6641029a0 0x7fa664197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664103360 0x7fa66419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410d420 con 0x7fa6641017b0 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410d2a0 con 0x7fa6641029a0 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa66b75b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410d5a0 con 0x7fa664103360 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49268/0 (socket says 172.21.15.38:49268) 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 -- 172.21.15.38:0/3869918714 learned_addr learned my addr 172.21.15.38:0/3869918714 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa669cf8640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664103360 0x7fa66419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa668cf6640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6641029a0 0x7fa664197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 -- 172.21.15.38:0/3869918714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664103360 msgr2=0x7fa66419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664103360 0x7fa66419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 -- 172.21.15.38:0/3869918714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6641029a0 msgr2=0x7fa664197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.061+0000 7fa6694f7640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6641029a0 0x7fa664197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa6694f7640 1 -- 172.21.15.38:0/3869918714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66419f6d0 con 0x7fa6641017b0 2024-04-22T01:10:17.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa6694f7640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664197ab0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c00 tx=0x7fa660005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:17.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660041070 con 0x7fa6641017b0 2024-04-22T01:10:17.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66419f960 con 0x7fa6641017b0 2024-04-22T01:10:17.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6600044d0 con 0x7fa6641017b0 2024-04-22T01:10:17.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660034440 con 0x7fa6641017b0 2024-04-22T01:10:17.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.062+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66419fea0 con 0x7fa6641017b0 2024-04-22T01:10:17.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.063+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa664108b70 con 0x7fa6641017b0 2024-04-22T01:10:17.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.064+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa660044070 con 0x7fa6641017b0 2024-04-22T01:10:17.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.065+0000 7fa65a7fc640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa6500a2b90 0x7fa6500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:17.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.065+0000 7fa668cf6640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa6500a2b90 0x7fa6500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:17.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.065+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6600f0870 con 0x7fa6641017b0 2024-04-22T01:10:17.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.065+0000 7fa668cf6640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa6500a2b90 0x7fa6500a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa664198fc0 tx=0x7fa654005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:17.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.068+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa6600bd620 con 0x7fa6641017b0 2024-04-22T01:10:17.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:16 smithi203 ceph-mon[31825]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:17.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.291+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa66402d170 con 0x7fa6641017b0 2024-04-22T01:10:17.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.291+0000 7fa65a7fc640 1 -- 172.21.15.38:0/3869918714 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fa6600dd030 con 0x7fa6641017b0 2024-04-22T01:10:17.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:17.291 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T01:10:17.291 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 9 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa6500a2b90 msgr2=0x7fa6500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa6500a2b90 0x7fa6500a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa664198fc0 tx=0x7fa654005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 msgr2=0x7fa664197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664197ab0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c00 tx=0x7fa660005d50 comp rx=0 tx=0).stop 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 shutdown_connections 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa6500a2b90 0x7fa6500a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664103360 0x7fa66419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6641029a0 0x7fa664197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 --2- 172.21.15.38:0/3869918714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6641017b0 0x7fa664197ab0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:17.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.293+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 >> 172.21.15.38:0/3869918714 conn(0x7fa6640fcd40 msgr2=0x7fa664105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:17.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.294+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 shutdown_connections 2024-04-22T01:10:17.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:17.294+0000 7fa66b75b640 1 -- 172.21.15.38:0/3869918714 wait complete. 2024-04-22T01:10:17.302 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:16 smithi038 ceph-mon[28043]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:16 smithi111 ceph-mon[31057]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3869918714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:18.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:18.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3869918714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:18.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3869918714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:19.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:18 smithi203 ceph-mon[31825]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:19.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:18 smithi203 ceph-mon[31825]: Deploying daemon mds.g on smithi038 2024-04-22T01:10:19.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:18 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] up:boot 2024-04-22T01:10:19.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:18 smithi203 ceph-mon[31825]: fsmap 9 up:standby 2024-04-22T01:10:19.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:18 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T01:10:19.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:19.327 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:18 smithi038 ceph-mon[28043]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:19.328 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:18 smithi038 ceph-mon[28043]: Deploying daemon mds.g on smithi038 2024-04-22T01:10:19.328 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:18 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] up:boot 2024-04-22T01:10:19.328 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:18 smithi038 ceph-mon[28043]: fsmap 9 up:standby 2024-04-22T01:10:19.328 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:18 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T01:10:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:18 smithi111 ceph-mon[31057]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:18 smithi111 ceph-mon[31057]: Deploying daemon mds.g on smithi038 2024-04-22T01:10:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:18 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] up:boot 2024-04-22T01:10:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:18 smithi111 ceph-mon[31057]: fsmap 9 up:standby 2024-04-22T01:10:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:18 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 -- 172.21.15.38:0/4092569061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c381006a0 msgr2=0x7f6c38100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/4092569061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c381006a0 0x7f6c38100ac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009900 tx=0x7f6c2802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 -- 172.21.15.38:0/4092569061 shutdown_connections 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/4092569061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c38102250 0x7f6c38108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/4092569061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38101890 0x7f6c38101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/4092569061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c381006a0 0x7f6c38100ac0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 -- 172.21.15.38:0/4092569061 >> 172.21.15.38:0/4092569061 conn(0x7f6c380fbc30 msgr2=0x7f6c380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 -- 172.21.15.38:0/4092569061 shutdown_connections 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.411+0000 7f6c3e97a640 1 -- 172.21.15.38:0/4092569061 wait complete. 2024-04-22T01:10:19.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 Processor -- start 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 -- start start 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c381006a0 0x7f6c3806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 0x7f6c3819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38102250 0x7f6c3819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3810c320 con 0x7f6c38101890 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3810c1a0 con 0x7f6c381006a0 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3e97a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3810c4a0 con 0x7f6c38102250 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 0x7f6c3819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 0x7f6c3819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49286/0 (socket says 172.21.15.38:49286) 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c377fe640 1 -- 172.21.15.38:0/2338060225 learned_addr learned my addr 172.21.15.38:0/2338060225 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c37fff640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c381006a0 0x7f6c3806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.412+0000 7f6c3cf17640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38102250 0x7f6c3819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c377fe640 1 -- 172.21.15.38:0/2338060225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38102250 msgr2=0x7f6c3819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c377fe640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38102250 0x7f6c3819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c377fe640 1 -- 172.21.15.38:0/2338060225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c381006a0 msgr2=0x7f6c3806a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:19.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c377fe640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c381006a0 0x7f6c3806a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c377fe640 1 -- 172.21.15.38:0/2338060225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3819f770 con 0x7f6c38101890 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c377fe640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 0x7f6c3819c840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00d770 tx=0x7f6c2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c017070 con 0x7f6c38101890 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c2c005500 con 0x7f6c38101890 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3819fa00 con 0x7f6c38101890 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c004e00 con 0x7f6c38101890 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.413+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3819ff50 con 0x7f6c38101890 2024-04-22T01:10:19.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.414+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c38107a40 con 0x7f6c38101890 2024-04-22T01:10:19.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.415+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6c2c004fa0 con 0x7f6c38101890 2024-04-22T01:10:19.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.416+0000 7f6c357fa640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c200a2c10 0x7f6c200a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:19.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.416+0000 7f6c37fff640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c200a2c10 0x7f6c200a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:19.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.416+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6c2c0ce560 con 0x7f6c38101890 2024-04-22T01:10:19.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.416+0000 7f6c37fff640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c200a2c10 0x7f6c200a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c28000c00 tx=0x7f6c28005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:19.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.420+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6c2c09b270 con 0x7f6c38101890 2024-04-22T01:10:19.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.645+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c38062340 con 0x7f6c38101890 2024-04-22T01:10:19.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.646+0000 7f6c357fa640 1 -- 172.21.15.38:0/2338060225 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f6c2c09f570 con 0x7f6c38101890 2024-04-22T01:10:19.645 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:19.645 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T01:10:19.645 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 10 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.647+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c200a2c10 msgr2=0x7f6c200a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.647+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c200a2c10 0x7f6c200a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c28000c00 tx=0x7f6c28005fd0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.647+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 msgr2=0x7f6c3819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.647+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 0x7f6c3819c840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00d770 tx=0x7f6c2c00dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 shutdown_connections 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c200a2c10 0x7f6c200a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38102250 0x7f6c3819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c38101890 0x7f6c3819c840 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 --2- 172.21.15.38:0/2338060225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c381006a0 0x7f6c3806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:19.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 >> 172.21.15.38:0/2338060225 conn(0x7f6c380fbc30 msgr2=0x7f6c38104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:19.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 shutdown_connections 2024-04-22T01:10:19.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:19.648+0000 7f6c3e97a640 1 -- 172.21.15.38:0/2338060225 wait complete. 2024-04-22T01:10:20.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:19 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2338060225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:20.327 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:19 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2338060225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:20.327 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 systemd[1]: Starting Ceph mds.g for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:10:20.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:19 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2338060225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:20.579 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 podman[68832]: 2024-04-22 01:10:20.23108854 +0000 UTC m=+0.018618639 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:20.579 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 podman[68832]: 2024-04-22 01:10:20.34019163 +0000 UTC m=+0.127721751 container create fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T01:10:20.579 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 podman[68832]: 2024-04-22 01:10:20.57662895 +0000 UTC m=+0.364159044 container init fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T01:10:20.579 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 podman[68832]: 2024-04-22 01:10:20.579511034 +0000 UTC m=+0.367041119 container start fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:10:20.905 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 ceph-mds[68851]: -- 172.21.15.38:0/2802291732 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55672997da40 con 0x556729951000 2024-04-22T01:10:20.905 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g[68847]: starting mds.g at 2024-04-22T01:10:20.905 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 bash[68832]: fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a 2024-04-22T01:10:20.905 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:10:20 smithi038 systemd[1]: Started Ceph mds.g for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:21.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:21.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:21 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:21 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:21.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 -- 172.21.15.38:0/729463547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 msgr2=0x7fc128100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 --2- 172.21.15.38:0/729463547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc128100ac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc118009900 tx=0x7fc11802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 -- 172.21.15.38:0/729463547 shutdown_connections 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 --2- 172.21.15.38:0/729463547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 0x7fc128108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 --2- 172.21.15.38:0/729463547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 0x7fc128101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 --2- 172.21.15.38:0/729463547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc128100ac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 -- 172.21.15.38:0/729463547 >> 172.21.15.38:0/729463547 conn(0x7fc1280fbc30 msgr2=0x7fc1280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 -- 172.21.15.38:0/729463547 shutdown_connections 2024-04-22T01:10:21.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.768+0000 7fc12c972640 1 -- 172.21.15.38:0/729463547 wait complete. 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 Processor -- start 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 -- start start 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc1281045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 0x7fc128102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 0x7fc1281031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc12810c360 con 0x7fc1281006a0 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc12810c1e0 con 0x7fc128102250 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12c972640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc12810c4e0 con 0x7fc128101890 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc1281045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc1281045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49328/0 (socket says 172.21.15.38:49328) 2024-04-22T01:10:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12659c640 1 -- 172.21.15.38:0/1894063951 learned_addr learned my addr 172.21.15.38:0/1894063951 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc125d9b640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 0x7fc128102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc126d9d640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 0x7fc1281031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12659c640 1 -- 172.21.15.38:0/1894063951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 msgr2=0x7fc128102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12659c640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 0x7fc128102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.769+0000 7fc12659c640 1 -- 172.21.15.38:0/1894063951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 msgr2=0x7fc1281031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc12659c640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 0x7fc1281031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc12659c640 1 -- 172.21.15.38:0/1894063951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1281039d0 con 0x7fc1281006a0 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc126d9d640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 0x7fc1281031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc125d9b640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 0x7fc128102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc12659c640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc1281045c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc11802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118043020 con 0x7fc1281006a0 2024-04-22T01:10:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1180054e0 con 0x7fc1281006a0 2024-04-22T01:10:21.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1281a1c10 con 0x7fc1281006a0 2024-04-22T01:10:21.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118003ce0 con 0x7fc1281006a0 2024-04-22T01:10:21.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.770+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1281a2130 con 0x7fc1281006a0 2024-04-22T01:10:21.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.771+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1281a2300 con 0x7fc1281006a0 2024-04-22T01:10:21.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.772+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc118046020 con 0x7fc1281006a0 2024-04-22T01:10:21.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.773+0000 7fc1177fe640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1100a2b90 0x7fc1100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:21.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.773+0000 7fc125d9b640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1100a2b90 0x7fc1100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:21.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.773+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc1180f11e0 con 0x7fc1281006a0 2024-04-22T01:10:21.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.773+0000 7fc125d9b640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1100a2b90 0x7fc1100a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0058c0 tx=0x7fc11c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:21.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:21.776+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc11804d020 con 0x7fc1281006a0 2024-04-22T01:10:21.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:21.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:21.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:21.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:21.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:21 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:21.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.000+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1281a24d0 con 0x7fc1281006a0 2024-04-22T01:10:22.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.001+0000 7fc1177fe640 1 -- 172.21.15.38:0/1894063951 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7fc1180be5e0 con 0x7fc1281006a0 2024-04-22T01:10:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:22.000 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14595,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T01:10:22.000 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 11 2024-04-22T01:10:22.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1100a2b90 msgr2=0x7fc1100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:22.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1100a2b90 0x7fc1100a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0058c0 tx=0x7fc11c005850 comp rx=0 tx=0).stop 2024-04-22T01:10:22.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 msgr2=0x7fc1281045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:22.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc1281045c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc11802c6c0 comp rx=0 tx=0).stop 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 shutdown_connections 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc1100a2b90 0x7fc1100a5050 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc128102250 0x7fc1281031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc128101890 0x7fc128102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 --2- 172.21.15.38:0/1894063951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1281006a0 0x7fc1281045c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 >> 172.21.15.38:0/1894063951 conn(0x7fc1280fbc30 msgr2=0x7fc1280fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 shutdown_connections 2024-04-22T01:10:22.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:22.003+0000 7fc12c972640 1 -- 172.21.15.38:0/1894063951 wait complete. 2024-04-22T01:10:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:22 smithi038 ceph-mon[28043]: Deploying daemon mds.a on smithi038 2024-04-22T01:10:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:22 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.38:6838/1149927748,v1:172.21.15.38:6839/1149927748] up:boot 2024-04-22T01:10:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:22 smithi038 ceph-mon[28043]: fsmap 10 up:standby 2024-04-22T01:10:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:22 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T01:10:22.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:22 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1894063951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:22 smithi203 ceph-mon[31825]: Deploying daemon mds.a on smithi038 2024-04-22T01:10:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:22 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.38:6838/1149927748,v1:172.21.15.38:6839/1149927748] up:boot 2024-04-22T01:10:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:22 smithi203 ceph-mon[31825]: fsmap 10 up:standby 2024-04-22T01:10:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:22 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T01:10:22.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:22 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1894063951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:22.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:22 smithi111 ceph-mon[31057]: Deploying daemon mds.a on smithi038 2024-04-22T01:10:22.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:22 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.38:6838/1149927748,v1:172.21.15.38:6839/1149927748] up:boot 2024-04-22T01:10:22.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:22 smithi111 ceph-mon[31057]: fsmap 10 up:standby 2024-04-22T01:10:22.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:22 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T01:10:22.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:22 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1894063951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:23.155 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:22 smithi038 systemd[1]: Starting Ceph mds.a for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:10:23.443 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 podman[69190]: 2024-04-22 01:10:23.085103951 +0000 UTC m=+0.018580243 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:10:23.444 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 podman[69190]: 2024-04-22 01:10:23.187995714 +0000 UTC m=+0.121472010 container create cccf62f2b37b0ae8338f8df1e7ea9ce5db62a3ea23f83f1d773c0d47fd650ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T01:10:23.444 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 podman[69190]: 2024-04-22 01:10:23.424250506 +0000 UTC m=+0.357726796 container init cccf62f2b37b0ae8338f8df1e7ea9ce5db62a3ea23f83f1d773c0d47fd650ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8) 2024-04-22T01:10:23.444 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 podman[69190]: 2024-04-22 01:10:23.427357099 +0000 UTC m=+0.360833389 container start cccf62f2b37b0ae8338f8df1e7ea9ce5db62a3ea23f83f1d773c0d47fd650ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8) 2024-04-22T01:10:23.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:23 smithi203 ceph-mon[31825]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:23.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:23 smithi111 ceph-mon[31057]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:23 smithi038 ceph-mon[28043]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:23.905 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 ceph-mds[69210]: -- 172.21.15.38:0/787214250 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55677244da40 con 0x556772421000 2024-04-22T01:10:23.905 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a[69206]: starting mds.a at 2024-04-22T01:10:23.905 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 bash[69190]: cccf62f2b37b0ae8338f8df1e7ea9ce5db62a3ea23f83f1d773c0d47fd650ad1 2024-04-22T01:10:23.905 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:10:23 smithi038 systemd[1]: Started Ceph mds.a for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:24.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 -- 172.21.15.38:0/4265339114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c81029a0 msgr2=0x7fd6c8102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/4265339114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c81029a0 0x7fd6c8102e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009900 tx=0x7fd6b002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 -- 172.21.15.38:0/4265339114 shutdown_connections 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/4265339114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6c8103360 0x7fd6c8109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/4265339114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c81029a0 0x7fd6c8102e20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/4265339114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6c81017b0 0x7fd6c8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 -- 172.21.15.38:0/4265339114 >> 172.21.15.38:0/4265339114 conn(0x7fd6c80fcd40 msgr2=0x7fd6c80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:24.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 -- 172.21.15.38:0/4265339114 shutdown_connections 2024-04-22T01:10:24.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.124+0000 7fd6ccb77640 1 -- 172.21.15.38:0/4265339114 wait complete. 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.125+0000 7fd6ccb77640 1 Processor -- start 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.125+0000 7fd6ccb77640 1 -- start start 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6c81017b0 0x7fd6c80754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6c81029a0 0x7fd6c8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 0x7fd6c8074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c810d430 con 0x7fd6c8103360 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c810d2b0 con 0x7fd6c81017b0 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c810d5b0 con 0x7fd6c81029a0 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 0x7fd6c8074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 0x7fd6c8074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49366/0 (socket says 172.21.15.38:49366) 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 -- 172.21.15.38:0/367150402 learned_addr learned my addr 172.21.15.38:0/367150402 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:24.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c659c640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6c81017b0 0x7fd6c80754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c5d9b640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6c81029a0 0x7fd6c8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 -- 172.21.15.38:0/367150402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6c81029a0 msgr2=0x7fd6c8073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6c81029a0 0x7fd6c8073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 -- 172.21.15.38:0/367150402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6c81017b0 msgr2=0x7fd6c80754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6c81017b0 0x7fd6c80754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 -- 172.21.15.38:0/367150402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c8074900 con 0x7fd6c8103360 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6c6d9d640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 0x7fd6c8074070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d750 tx=0x7fd6c000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0017070 con 0x7fd6c8103360 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c0005500 con 0x7fd6c8103360 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0004d90 con 0x7fd6c8103360 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c81a1ba0 con 0x7fd6c8103360 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.126+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c81a20c0 con 0x7fd6c8103360 2024-04-22T01:10:24.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.127+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c8108ad0 con 0x7fd6c8103360 2024-04-22T01:10:24.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.128+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6c0004f30 con 0x7fd6c8103360 2024-04-22T01:10:24.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.129+0000 7fd6bf7fe640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6b40a2c10 0x7fd6b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:24.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.129+0000 7fd6c659c640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6b40a2c10 0x7fd6b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:24.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.130+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6c00ce560 con 0x7fd6c8103360 2024-04-22T01:10:24.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.130+0000 7fd6c659c640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6b40a2c10 0x7fd6b40a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8102800 tx=0x7fd6b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:24.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.134+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd6c009b8c0 con 0x7fd6c8103360 2024-04-22T01:10:24.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.355+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6c80623e0 con 0x7fd6c8103360 2024-04-22T01:10:24.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.355+0000 7fd6bf7fe640 1 -- 172.21.15.38:0/367150402 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7fd6c009b1c0 con 0x7fd6c8103360 2024-04-22T01:10:24.355 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:24.355 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14595,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T01:10:24.355 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 11 2024-04-22T01:10:24.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6b40a2c10 msgr2=0x7fd6b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6b40a2c10 0x7fd6b40a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8102800 tx=0x7fd6b8009500 comp rx=0 tx=0).stop 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 msgr2=0x7fd6c8074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 0x7fd6c8074070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d750 tx=0x7fd6c000dc10 comp rx=0 tx=0).stop 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 shutdown_connections 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6b40a2c10 0x7fd6b40a50d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6c8103360 0x7fd6c8074070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6c81029a0 0x7fd6c8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 --2- 172.21.15.38:0/367150402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6c81017b0 0x7fd6c80754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:24.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 >> 172.21.15.38:0/367150402 conn(0x7fd6c80fcd40 msgr2=0x7fd6c8105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:24.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 shutdown_connections 2024-04-22T01:10:24.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:24.357+0000 7fd6ccb77640 1 -- 172.21.15.38:0/367150402 wait complete. 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:24.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: Deploying daemon mds.b on smithi111 2024-04-22T01:10:24.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/367150402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:24.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.38:6840/3631879100,v1:172.21.15.38:6841/3631879100] up:boot 2024-04-22T01:10:24.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: fsmap 11 up:standby 2024-04-22T01:10:24.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:24 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T01:10:24.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: Deploying daemon mds.b on smithi111 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/367150402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.38:6840/3631879100,v1:172.21.15.38:6841/3631879100] up:boot 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: fsmap 11 up:standby 2024-04-22T01:10:24.898 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:24 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: Deploying daemon mds.b on smithi111 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/367150402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.38:6840/3631879100,v1:172.21.15.38:6841/3631879100] up:boot 2024-04-22T01:10:25.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: fsmap 11 up:standby 2024-04-22T01:10:25.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:24 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T01:10:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:25 smithi111 ceph-mon[31057]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:25.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:25 smithi038 ceph-mon[28043]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:26.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:25 smithi203 ceph-mon[31825]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:26.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/1376006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf0103360 msgr2=0x7f5cf0109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/1376006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf0103360 0x7f5cf0109bf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009ca0 tx=0x7f5ce802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/1376006271 shutdown_connections 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/1376006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf0103360 0x7f5cf0109bf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/1376006271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cf01029a0 0x7f5cf0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/1376006271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cf01017b0 0x7f5cf0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/1376006271 >> 172.21.15.38:0/1376006271 conn(0x7f5cf00fcd40 msgr2=0x7f5cf00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:26.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/1376006271 shutdown_connections 2024-04-22T01:10:26.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.485+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/1376006271 wait complete. 2024-04-22T01:10:26.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.486+0000 7f5cf4ab5640 1 Processor -- start 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.486+0000 7f5cf4ab5640 1 -- start start 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.486+0000 7f5cf4ab5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cf01017b0 0x7f5cf0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.486+0000 7f5cf4ab5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 0x7f5cf019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.486+0000 7f5cf4ab5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cf0103360 0x7f5cf01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5cf4ab5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf010d430 con 0x7f5cf01029a0 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5cf4ab5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf010d2b0 con 0x7f5cf0103360 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5cf4ab5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf010d5b0 con 0x7f5cf01017b0 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 0x7f5cf019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 0x7f5cf019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49382/0 (socket says 172.21.15.38:49382) 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 -- 172.21.15.38:0/922736499 learned_addr learned my addr 172.21.15.38:0/922736499 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:26.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ceed9d640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cf0103360 0x7f5cf01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5cee59c640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cf01017b0 0x7f5cf0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 -- 172.21.15.38:0/922736499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cf01017b0 msgr2=0x7f5cf0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cf01017b0 0x7f5cf0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 -- 172.21.15.38:0/922736499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cf0103360 msgr2=0x7f5cf01a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cf0103360 0x7f5cf01a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 -- 172.21.15.38:0/922736499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf01a18d0 con 0x7f5cf01029a0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce5d9b640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 0x7f5cf019a1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800d770 tx=0x7f5cd800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8017070 con 0x7f5cf01029a0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf01a1bc0 con 0x7f5cf01029a0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd8005500 con 0x7f5cf01029a0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8004e00 con 0x7f5cf01029a0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.487+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf01a20c0 con 0x7f5cf01029a0 2024-04-22T01:10:26.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.488+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf01a2290 con 0x7f5cf01029a0 2024-04-22T01:10:26.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.490+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5cd8004fa0 con 0x7f5cf01029a0 2024-04-22T01:10:26.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.490+0000 7f5ce7fff640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5cdc0a2b90 0x7f5cdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.490+0000 7f5cee59c640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5cdc0a2b90 0x7f5cdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.490+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5cd80ce6d0 con 0x7f5cf01029a0 2024-04-22T01:10:26.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.491+0000 7f5cee59c640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5cdc0a2b90 0x7f5cdc0a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5cf019ab60 tx=0x7f5ce0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:26.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.495+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5cd809bad0 con 0x7f5cf01029a0 2024-04-22T01:10:26.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.723+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cf01a2500 con 0x7f5cf01029a0 2024-04-22T01:10:26.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.724+0000 7f5ce7fff640 1 -- 172.21.15.38:0/922736499 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10494 (secure 0 0 0) 0x7f5cd809b3d0 con 0x7f5cf01029a0 2024-04-22T01:10:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:26.723 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14595,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24527,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-22T01:10:26.724 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 13 2024-04-22T01:10:26.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5cdc0a2b90 msgr2=0x7f5cdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5cdc0a2b90 0x7f5cdc0a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5cf019ab60 tx=0x7f5ce0005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:26.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 msgr2=0x7f5cf019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 0x7f5cf019a1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800d770 tx=0x7f5cd800dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 shutdown_connections 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5cdc0a2b90 0x7f5cdc0a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cf0103360 0x7f5cf01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cf01029a0 0x7f5cf019a1d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 --2- 172.21.15.38:0/922736499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cf01017b0 0x7f5cf0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 >> 172.21.15.38:0/922736499 conn(0x7f5cf00fcd40 msgr2=0x7f5cf0073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 shutdown_connections 2024-04-22T01:10:26.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.726+0000 7f5cf4ab5640 1 -- 172.21.15.38:0/922736499 wait complete. 2024-04-22T01:10:26.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:26.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 -- 172.21.15.38:0/555878196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 msgr2=0x7fc750107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 --2- 172.21.15.38:0/555878196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750107000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc738009900 tx=0x7fc73802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 -- 172.21.15.38:0/555878196 shutdown_connections 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 --2- 172.21.15.38:0/555878196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc750107540 0x7fc750109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 --2- 172.21.15.38:0/555878196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750107000 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 --2- 172.21.15.38:0/555878196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7501022e0 0x7fc7501046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 -- 172.21.15.38:0/555878196 >> 172.21.15.38:0/555878196 conn(0x7fc7500fbc30 msgr2=0x7fc7500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 -- 172.21.15.38:0/555878196 shutdown_connections 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.843+0000 7fc755a27640 1 -- 172.21.15.38:0/555878196 wait complete. 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 Processor -- start 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 -- start start 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7501022e0 0x7fc750197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc750107540 0x7fc75019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc75010d400 con 0x7fc750104c10 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc75010d280 con 0x7fc750107540 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc755a27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc75010d580 con 0x7fc7501022e0 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc7467fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc7467fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38812/0 (socket says 172.21.15.38:38812) 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc7467fc640 1 -- 172.21.15.38:0/4034183730 learned_addr learned my addr 172.21.15.38:0/4034183730 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc74effd640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7501022e0 0x7fc750197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.844+0000 7fc74f7fe640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc750107540 0x7fc75019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc7467fc640 1 -- 172.21.15.38:0/4034183730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7501022e0 msgr2=0x7fc750197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc7467fc640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7501022e0 0x7fc750197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc7467fc640 1 -- 172.21.15.38:0/4034183730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc750107540 msgr2=0x7fc75019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc7467fc640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc750107540 0x7fc75019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:26.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc7467fc640 1 -- 172.21.15.38:0/4034183730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75019f730 con 0x7fc750104c10 2024-04-22T01:10:26.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc7467fc640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750198050 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc738000c00 tx=0x7fc738005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:26.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738041070 con 0x7fc750104c10 2024-04-22T01:10:26.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75019f9c0 con 0x7fc750104c10 2024-04-22T01:10:26.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7380044d0 con 0x7fc750104c10 2024-04-22T01:10:26.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738034440 con 0x7fc750104c10 2024-04-22T01:10:26.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.845+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75019ff00 con 0x7fc750104c10 2024-04-22T01:10:26.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.846+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7500fd720 con 0x7fc750104c10 2024-04-22T01:10:26.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.848+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc738044070 con 0x7fc750104c10 2024-04-22T01:10:26.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.848+0000 7fc754a25640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc73c0a2b90 0x7fc73c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:26.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.849+0000 7fc74effd640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc73c0a2b90 0x7fc73c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:26.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.849+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc7380f0910 con 0x7fc750104c10 2024-04-22T01:10:26.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.849+0000 7fc74effd640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc73c0a2b90 0x7fc73c0a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc740005d00 tx=0x7fc74000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:26.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:26.852+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc7380bd620 con 0x7fc750104c10 2024-04-22T01:10:26.895 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:10:26 smithi111 systemd[1]: Started Ceph mds.b for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:10:27.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.040+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc75002d3c0 con 0x7fc750104c10 2024-04-22T01:10:27.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.041+0000 7fc754a25640 1 -- 172.21.15.38:0/4034183730 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18491 (secure 0 0 0) 0x7fc7380dd070 con 0x7fc750104c10 2024-04-22T01:10:27.041 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:27.041 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:09:40.128137+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc73c0a2b90 msgr2=0x7fc73c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc73c0a2b90 0x7fc73c0a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc740005d00 tx=0x7fc74000b410 comp rx=0 tx=0).stop 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 msgr2=0x7fc750198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750198050 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc738000c00 tx=0x7fc738005d50 comp rx=0 tx=0).stop 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 shutdown_connections 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc73c0a2b90 0x7fc73c0a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc750107540 0x7fc75019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc750104c10 0x7fc750198050 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 --2- 172.21.15.38:0/4034183730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7501022e0 0x7fc750197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.043+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 >> 172.21.15.38:0/4034183730 conn(0x7fc7500fbc30 msgr2=0x7fc750102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:27.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.044+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 shutdown_connections 2024-04-22T01:10:27.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.044+0000 7fc755a27640 1 -- 172.21.15.38:0/4034183730 wait complete. 2024-04-22T01:10:27.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T01:10:27.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:10:27.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 -- 172.21.15.38:0/3256927111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8102200 msgr2=0x7f7ec81045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 --2- 172.21.15.38:0/3256927111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8102200 0x7f7ec81045f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009900 tx=0x7f7eb802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 -- 172.21.15.38:0/3256927111 shutdown_connections 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 --2- 172.21.15.38:0/3256927111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ec8107460 0x7f7ec81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 --2- 172.21.15.38:0/3256927111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8104b30 0x7f7ec8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 --2- 172.21.15.38:0/3256927111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8102200 0x7f7ec81045f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 -- 172.21.15.38:0/3256927111 >> 172.21.15.38:0/3256927111 conn(0x7f7ec80fbbf0 msgr2=0x7f7ec80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 -- 172.21.15.38:0/3256927111 shutdown_connections 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.161+0000 7f7ed010f640 1 -- 172.21.15.38:0/3256927111 wait complete. 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 Processor -- start 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 -- start start 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8102200 0x7f7ec8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ec8104b30 0x7f7ec8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 0x7f7ec819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec810d430 con 0x7f7ec8107460 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec810d2b0 con 0x7f7ec8104b30 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ed010f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec810d5b0 con 0x7f7ec8102200 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 0x7f7ec819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 0x7f7ec819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38822/0 (socket says 172.21.15.38:38822) 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 -- 172.21.15.38:0/1061757239 learned_addr learned my addr 172.21.15.38:0/1061757239 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ecdeab640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8102200 0x7f7ec8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ecd6aa640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ec8104b30 0x7f7ec8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 -- 172.21.15.38:0/1061757239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8102200 msgr2=0x7f7ec8197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8102200 0x7f7ec8197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 -- 172.21.15.38:0/1061757239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ec8104b30 msgr2=0x7f7ec8198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ec8104b30 0x7f7ec8198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.162+0000 7f7ece6ac640 1 -- 172.21.15.38:0/1061757239 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec819f760 con 0x7f7ec8107460 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.163+0000 7f7ece6ac640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 0x7f7ec819f0f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000eb10 tx=0x7f7ec000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.163+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0004b20 con 0x7f7ec8107460 2024-04-22T01:10:27.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.163+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ec0004cc0 con 0x7f7ec8107460 2024-04-22T01:10:27.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.163+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec8100050 con 0x7f7ec8107460 2024-04-22T01:10:27.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.163+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0010760 con 0x7f7ec8107460 2024-04-22T01:10:27.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.163+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec8100550 con 0x7f7ec8107460 2024-04-22T01:10:27.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.164+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec80fd6e0 con 0x7f7ec8107460 2024-04-22T01:10:27.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.165+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7ec0005000 con 0x7f7ec8107460 2024-04-22T01:10:27.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.166+0000 7f7ebeffd640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7eb40a2b90 0x7f7eb40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.166+0000 7f7ecdeab640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7eb40a2b90 0x7f7eb40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.166+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7ec00cf3e0 con 0x7f7ec8107460 2024-04-22T01:10:27.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.166+0000 7f7ecdeab640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7eb40a2b90 0x7f7eb40a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7eb802c7b0 tx=0x7f7eb8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:27.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.169+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7ec00536f0 con 0x7f7ec8107460 2024-04-22T01:10:27.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.369+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7ec802d170 con 0x7f7ec8107460 2024-04-22T01:10:27.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.369+0000 7f7ebeffd640 1 -- 172.21.15.38:0/1061757239 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f7ec009c740 con 0x7f7ec8107460 2024-04-22T01:10:27.369 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7eb40a2b90 msgr2=0x7f7eb40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7eb40a2b90 0x7f7eb40a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7eb802c7b0 tx=0x7f7eb8005fd0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 msgr2=0x7f7ec819f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 0x7f7ec819f0f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000eb10 tx=0x7f7ec000c2d0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 shutdown_connections 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7eb40a2b90 0x7f7eb40a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ec8107460 0x7f7ec819f0f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ec8104b30 0x7f7ec8198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 --2- 172.21.15.38:0/1061757239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ec8102200 0x7f7ec8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 >> 172.21.15.38:0/1061757239 conn(0x7f7ec80fbbf0 msgr2=0x7f7ec8102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.371+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 shutdown_connections 2024-04-22T01:10:27.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.372+0000 7f7ed010f640 1 -- 172.21.15.38:0/1061757239 wait complete. 2024-04-22T01:10:27.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T01:10:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 -- 172.21.15.38:0/501536877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c102250 msgr2=0x7fc29c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/501536877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c102250 0x7fc29c108ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc294009ca0 tx=0x7fc29402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 -- 172.21.15.38:0/501536877 shutdown_connections 2024-04-22T01:10:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/501536877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c102250 0x7fc29c108ae0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/501536877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c101890 0x7fc29c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/501536877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc29c1006a0 0x7fc29c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 -- 172.21.15.38:0/501536877 >> 172.21.15.38:0/501536877 conn(0x7fc29c0fbc30 msgr2=0x7fc29c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 -- 172.21.15.38:0/501536877 shutdown_connections 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.486+0000 7fc2a23a7640 1 -- 172.21.15.38:0/501536877 wait complete. 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 Processor -- start 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 -- start start 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 0x7fc29c104730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc29c101890 0x7fc29c102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c102250 0x7fc29c103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c10c300 con 0x7fc29c1006a0 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c10c180 con 0x7fc29c102250 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a23a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c10c480 con 0x7fc29c101890 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc29bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 0x7fc29c104730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc29bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 0x7fc29c104730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38834/0 (socket says 172.21.15.38:38834) 2024-04-22T01:10:27.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc29bfff640 1 -- 172.21.15.38:0/2109306826 learned_addr learned my addr 172.21.15.38:0/2109306826 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc29b7fe640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc29c101890 0x7fc29c102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.487+0000 7fc2a0944640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c102250 0x7fc29c103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc29bfff640 1 -- 172.21.15.38:0/2109306826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc29c101890 msgr2=0x7fc29c102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc29bfff640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc29c101890 0x7fc29c102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc29bfff640 1 -- 172.21.15.38:0/2109306826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c102250 msgr2=0x7fc29c103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc29bfff640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c102250 0x7fc29c103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc29bfff640 1 -- 172.21.15.38:0/2109306826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29c103b50 con 0x7fc29c1006a0 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc2a0944640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c102250 0x7fc29c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc29bfff640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 0x7fc29c104730 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00d770 tx=0x7fc28c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c017070 con 0x7fc29c1006a0 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc28c009c30 con 0x7fc29c1006a0 2024-04-22T01:10:27.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc29c1a1ba0 con 0x7fc29c1006a0 2024-04-22T01:10:27.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c003da0 con 0x7fc29c1006a0 2024-04-22T01:10:27.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.488+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc29c1a20c0 con 0x7fc29c1006a0 2024-04-22T01:10:27.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.489+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc29c1a2290 con 0x7fc29c1006a0 2024-04-22T01:10:27.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.490+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc28c003f40 con 0x7fc29c1006a0 2024-04-22T01:10:27.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.491+0000 7fc2997fa640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc2840a2b90 0x7fc2840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.491+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc28c0ce910 con 0x7fc29c1006a0 2024-04-22T01:10:27.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.491+0000 7fc29b7fe640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc2840a2b90 0x7fc2840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.491+0000 7fc29b7fe640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc2840a2b90 0x7fc2840a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc29c1044a0 tx=0x7fc29000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:27.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.494+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc28c09bd10 con 0x7fc29c1006a0 2024-04-22T01:10:27.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:27.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] up:boot 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: fsmap 12 up:standby 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/922736499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4034183730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:27 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1061757239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:27.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.685+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc29c1a2640 con 0x7fc29c1006a0 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.686+0000 7fc2997fa640 1 -- 172.21.15.38:0/2109306826 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fc28c009de0 con 0x7fc29c1006a0 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout: "poolnum": 1, 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout: "poolname": ".mgr" 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:10:27.685 INFO:teuthology.orchestra.run.smithi038.stdout:] 2024-04-22T01:10:27.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.687+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc2840a2b90 msgr2=0x7fc2840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.687+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc2840a2b90 0x7fc2840a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc29c1044a0 tx=0x7fc29000a2d0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 msgr2=0x7fc29c104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 0x7fc29c104730 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00d770 tx=0x7fc28c00dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:27.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 shutdown_connections 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc2840a2b90 0x7fc2840a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc29c102250 0x7fc29c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc29c101890 0x7fc29c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 --2- 172.21.15.38:0/2109306826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc29c1006a0 0x7fc29c104730 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 >> 172.21.15.38:0/2109306826 conn(0x7fc29c0fbc30 msgr2=0x7fc29c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 shutdown_connections 2024-04-22T01:10:27.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.688+0000 7fc2a23a7640 1 -- 172.21.15.38:0/2109306826 wait complete. 2024-04-22T01:10:27.698 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T01:10:27.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 -- 172.21.15.38:0/179532230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 msgr2=0x7f4ca8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 --2- 172.21.15.38:0/179532230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8100ac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009ca0 tx=0x7f4c9802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 -- 172.21.15.38:0/179532230 shutdown_connections 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 --2- 172.21.15.38:0/179532230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8102250 0x7f4ca8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 --2- 172.21.15.38:0/179532230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8101890 0x7f4ca8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 --2- 172.21.15.38:0/179532230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8100ac0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 -- 172.21.15.38:0/179532230 >> 172.21.15.38:0/179532230 conn(0x7f4ca80fbc30 msgr2=0x7f4ca80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 -- 172.21.15.38:0/179532230 shutdown_connections 2024-04-22T01:10:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.808+0000 7f4cacb97640 1 -- 172.21.15.38:0/179532230 wait complete. 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 Processor -- start 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 -- start start 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8101890 0x7f4ca8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102250 0x7f4ca819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca810c320 con 0x7f4ca81006a0 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca810c1a0 con 0x7f4ca8102250 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4cacb97640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca810c4a0 con 0x7f4ca8101890 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4ca659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4ca659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38842/0 (socket says 172.21.15.38:38842) 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.809+0000 7f4ca659c640 1 -- 172.21.15.38:0/4257025917 learned_addr learned my addr 172.21.15.38:0/4257025917 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca6d9d640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102250 0x7f4ca819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca5d9b640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8101890 0x7f4ca8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca659c640 1 -- 172.21.15.38:0/4257025917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8101890 msgr2=0x7f4ca8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca659c640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8101890 0x7f4ca8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca659c640 1 -- 172.21.15.38:0/4257025917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102250 msgr2=0x7f4ca819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca659c640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102250 0x7f4ca819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca659c640 1 -- 172.21.15.38:0/4257025917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca819f720 con 0x7f4ca81006a0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca5d9b640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8101890 0x7f4ca8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4ca659c640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8197b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000f80 tx=0x7f4c98032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98041020 con 0x7f4ca81006a0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca819f9b0 con 0x7f4ca81006a0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c98034cd0 con 0x7f4ca81006a0 2024-04-22T01:10:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98039040 con 0x7f4ca81006a0 2024-04-22T01:10:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.810+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca819fec0 con 0x7f4ca81006a0 2024-04-22T01:10:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.811+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca81a00f0 con 0x7f4ca81006a0 2024-04-22T01:10:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.812+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4c98044020 con 0x7f4ca81006a0 2024-04-22T01:10:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.813+0000 7f4c9f7fe640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c940a2b90 0x7f4c940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.813+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4c980f0f50 con 0x7f4ca81006a0 2024-04-22T01:10:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.813+0000 7f4ca5d9b640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c940a2b90 0x7f4c940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.813+0000 7f4ca5d9b640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c940a2b90 0x7f4c940a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8199010 tx=0x7f4c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:27.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:27.816+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4c9804b020 con 0x7f4ca81006a0 2024-04-22T01:10:27.817 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:27.817 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] up:boot 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: fsmap 12 up:standby 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/922736499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4034183730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.818 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:27 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1061757239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] up:boot 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: fsmap 12 up:standby 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/922736499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4034183730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:27.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:27 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1061757239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:28.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.002+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4ca81a0420 con 0x7f4ca81006a0 2024-04-22T01:10:28.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.563+0000 7f4c9f7fe640 1 -- 172.21.15.38:0/4257025917 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f4c980be3d0 con 0x7f4ca81006a0 2024-04-22T01:10:28.563 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' created 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c940a2b90 msgr2=0x7f4c940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c940a2b90 0x7f4c940a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8199010 tx=0x7f4c90005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 msgr2=0x7f4ca8197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8197b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000f80 tx=0x7f4c98032ec0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 shutdown_connections 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c940a2b90 0x7f4c940a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102250 0x7f4ca819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8101890 0x7f4ca8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.565+0000 7f4cacb97640 1 --2- 172.21.15.38:0/4257025917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca81006a0 0x7f4ca8197b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.566+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 >> 172.21.15.38:0/4257025917 conn(0x7f4ca80fbc30 msgr2=0x7f4ca80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:28.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.566+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 shutdown_connections 2024-04-22T01:10:28.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.566+0000 7f4cacb97640 1 -- 172.21.15.38:0/4257025917 wait complete. 2024-04-22T01:10:28.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 -- 172.21.15.38:0/318236587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4101850 msgr2=0x7effb4101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 --2- 172.21.15.38:0/318236587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4101850 0x7effb4101cd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7effa4009900 tx=0x7effa402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 -- 172.21.15.38:0/318236587 shutdown_connections 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 --2- 172.21.15.38:0/318236587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4102210 0x7effb4108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 --2- 172.21.15.38:0/318236587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4101850 0x7effb4101cd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 --2- 172.21.15.38:0/318236587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effb4100660 0x7effb4100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 -- 172.21.15.38:0/318236587 >> 172.21.15.38:0/318236587 conn(0x7effb40fbbd0 msgr2=0x7effb40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 -- 172.21.15.38:0/318236587 shutdown_connections 2024-04-22T01:10:28.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.692+0000 7effbb555640 1 -- 172.21.15.38:0/318236587 wait complete. 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 Processor -- start 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 -- start start 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 0x7effb4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4101850 0x7effb4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effb4102210 0x7effb419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7effb410d400 con 0x7effb4100660 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7effb410d280 con 0x7effb4102210 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effbb555640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7effb410d580 con 0x7effb4101850 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 0x7effb4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 0x7effb4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38852/0 (socket says 172.21.15.38:38852) 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 -- 172.21.15.38:0/2739992022 learned_addr learned my addr 172.21.15.38:0/2739992022 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:28.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb8af0640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4101850 0x7effb4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 -- 172.21.15.38:0/2739992022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4101850 msgr2=0x7effb4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4101850 0x7effb4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 -- 172.21.15.38:0/2739992022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effb4102210 msgr2=0x7effb419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb9af2640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effb4102210 0x7effb419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effb4102210 0x7effb419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb92f1640 1 -- 172.21.15.38:0/2739992022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb419f6a0 con 0x7effb4100660 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.693+0000 7effb8af0640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4101850 0x7effb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.694+0000 7effb92f1640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 0x7effb4197a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7effb000d770 tx=0x7effb000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.694+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0017070 con 0x7effb4100660 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.694+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effb0009c30 con 0x7effb4100660 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.694+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb419f990 con 0x7effb4100660 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.694+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0003dd0 con 0x7effb4100660 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.694+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb419feb0 con 0x7effb4100660 2024-04-22T01:10:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.695+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4107980 con 0x7effb4100660 2024-04-22T01:10:28.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.696+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7effb0003fb0 con 0x7effb4100660 2024-04-22T01:10:28.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.697+0000 7effaa7fc640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effa00a2b90 0x7effa00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:28.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.697+0000 7effb8af0640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effa00a2b90 0x7effa00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:28.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.697+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7effb00cec00 con 0x7effb4100660 2024-04-22T01:10:28.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.697+0000 7effb8af0640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effa00a2b90 0x7effa00a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7effa4006010 tx=0x7effa402c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:28.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.700+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7effb0009de0 con 0x7effb4100660 2024-04-22T01:10:28.709 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:28 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2109306826' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T01:10:28.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:28 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4257025917' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T01:10:28.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:28 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:28.893+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7effb402d170 con 0x7effb4100660 2024-04-22T01:10:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:28 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2109306826' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T01:10:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:28 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4257025917' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T01:10:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:28 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:28 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2109306826' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T01:10:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:28 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4257025917' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T01:10:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:28 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.569+0000 7effaa7fc640 1 -- 172.21.15.38:0/2739992022 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7effb009bdf0 con 0x7effb4100660 2024-04-22T01:10:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' created 2024-04-22T01:10:29.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effa00a2b90 msgr2=0x7effa00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:29.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effa00a2b90 0x7effa00a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7effa4006010 tx=0x7effa402c8a0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 msgr2=0x7effb4197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:29.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 0x7effb4197a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7effb000d770 tx=0x7effb000dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:29.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 shutdown_connections 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effa00a2b90 0x7effa00a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effb4102210 0x7effb419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb4101850 0x7effb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 --2- 172.21.15.38:0/2739992022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effb4100660 0x7effb4197a80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 >> 172.21.15.38:0/2739992022 conn(0x7effb40fbbd0 msgr2=0x7effb4104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 shutdown_connections 2024-04-22T01:10:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.572+0000 7effbb555640 1 -- 172.21.15.38:0/2739992022 wait complete. 2024-04-22T01:10:29.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T01:10:29.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.706+0000 7f21d944e640 1 -- 172.21.15.38:0/1448773428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 msgr2=0x7f21d4104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.706+0000 7f21d944e640 1 --2- 172.21.15.38:0/1448773428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4104600 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009900 tx=0x7f21c402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- 172.21.15.38:0/1448773428 shutdown_connections 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 --2- 172.21.15.38:0/1448773428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d4107470 0x7f21d41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 --2- 172.21.15.38:0/1448773428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 0x7f21d4106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 --2- 172.21.15.38:0/1448773428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4104600 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- 172.21.15.38:0/1448773428 >> 172.21.15.38:0/1448773428 conn(0x7f21d40fbc00 msgr2=0x7f21d40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- 172.21.15.38:0/1448773428 shutdown_connections 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- 172.21.15.38:0/1448773428 wait complete. 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 Processor -- start 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- start start 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 0x7f21d4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d4107470 0x7f21d419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:29.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d480 con 0x7f21d4102210 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d300 con 0x7f21d4104b40 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.707+0000 7f21d944e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d600 con 0x7f21d4107470 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38880/0 (socket says 172.21.15.38:38880) 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 -- 172.21.15.38:0/730857694 learned_addr learned my addr 172.21.15.38:0/730857694 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d27fc640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 0x7f21d4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d37fe640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d4107470 0x7f21d419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 -- 172.21.15.38:0/730857694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d4107470 msgr2=0x7f21d419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d4107470 0x7f21d419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 -- 172.21.15.38:0/730857694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 msgr2=0x7f21d4198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 0x7f21d4198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 -- 172.21.15.38:0/730857694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d419f720 con 0x7f21d4102210 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d27fc640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 0x7f21d4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d2ffd640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4197b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005ab0 tx=0x7f21c402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4043020 con 0x7f21d4102210 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d419f9b0 con 0x7f21d4102210 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c40054e0 con 0x7f21d4102210 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4003ce0 con 0x7f21d4102210 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.708+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d419ff00 con 0x7f21d4102210 2024-04-22T01:10:29.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.709+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d40fd6f0 con 0x7f21d4102210 2024-04-22T01:10:29.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.711+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f21c4046020 con 0x7f21d4102210 2024-04-22T01:10:29.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.711+0000 7f21bbfff640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21bc0a2b90 0x7f21bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:29.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.711+0000 7f21d27fc640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21bc0a2b90 0x7f21bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:29.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.711+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f21c40f1360 con 0x7f21d4102210 2024-04-22T01:10:29.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.712+0000 7f21d27fc640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21bc0a2b90 0x7f21bc0a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21d4199010 tx=0x7f21c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:29.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.715+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f21c404d020 con 0x7f21d4102210 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4257025917' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:10:29.797 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:29 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2739992022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-22T01:10:29.841 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:29.841 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4257025917' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T01:10:29.841 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:10:29.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:29 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2739992022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-22T01:10:29.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:29.942+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f21d402d290 con 0x7f21d4102210 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4257025917' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:10:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:10:30.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:29 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2739992022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-22T01:10:30.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.592+0000 7f21bbfff640 1 -- 172.21.15.38:0/730857694 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f21d402d290 con 0x7f21d4102210 2024-04-22T01:10:30.592 INFO:teuthology.orchestra.run.smithi038.stderr:new fs with metadata pool 2 and data pool 3 2024-04-22T01:10:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21bc0a2b90 msgr2=0x7f21bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21bc0a2b90 0x7f21bc0a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21d4199010 tx=0x7f21c8005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 msgr2=0x7f21d4197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4197b00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005ab0 tx=0x7f21c402c6c0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 shutdown_connections 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21bc0a2b90 0x7f21bc0a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d4107470 0x7f21d419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d4104b40 0x7f21d4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 --2- 172.21.15.38:0/730857694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21d4102210 0x7f21d4197b00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 >> 172.21.15.38:0/730857694 conn(0x7f21d40fbc00 msgr2=0x7f21d4102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 shutdown_connections 2024-04-22T01:10:30.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.595+0000 7f21d944e640 1 -- 172.21.15.38:0/730857694 wait complete. 2024-04-22T01:10:30.608 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-22T01:10:30.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-22T01:10:30.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 -- 172.21.15.38:0/425898005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 msgr2=0x7f22f0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:30.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 --2- 172.21.15.38:0/425898005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0101bc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009900 tx=0x7f22ec02bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:30.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 -- 172.21.15.38:0/425898005 shutdown_connections 2024-04-22T01:10:30.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 --2- 172.21.15.38:0/425898005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 0x7f22f0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 --2- 172.21.15.38:0/425898005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0102990 0x7f22f0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 --2- 172.21.15.38:0/425898005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0101bc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.716+0000 7f22f824b640 1 -- 172.21.15.38:0/425898005 >> 172.21.15.38:0/425898005 conn(0x7f22f00fcd30 msgr2=0x7f22f00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 -- 172.21.15.38:0/425898005 shutdown_connections 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 -- 172.21.15.38:0/425898005 wait complete. 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 Processor -- start 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 -- start start 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0102990 0x7f22f0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 0x7f22f019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22f010d410 con 0x7f22f01017a0 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22f010d290 con 0x7f22f0103350 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f824b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22f010d590 con 0x7f22f0102990 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f5fe7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f5fe7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38894/0 (socket says 172.21.15.38:38894) 2024-04-22T01:10:30.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f5fe7640 1 -- 172.21.15.38:0/3247468289 learned_addr learned my addr 172.21.15.38:0/3247468289 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.717+0000 7f22f67e8640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 0x7f22f019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f57e6640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0102990 0x7f22f0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f5fe7640 1 -- 172.21.15.38:0/3247468289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0102990 msgr2=0x7f22f0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f5fe7640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0102990 0x7f22f0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f5fe7640 1 -- 172.21.15.38:0/3247468289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 msgr2=0x7f22f019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f5fe7640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 0x7f22f019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f5fe7640 1 -- 172.21.15.38:0/3247468289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f019f6a0 con 0x7f22f01017a0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f67e8640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 0x7f22f019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f5fe7640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0197a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22ec000c00 tx=0x7f22ec02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec041070 con 0x7f22f01017a0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22ec004450 con 0x7f22f01017a0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f019f930 con 0x7f22f01017a0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec034440 con 0x7f22f01017a0 2024-04-22T01:10:30.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.718+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f019fe70 con 0x7f22f01017a0 2024-04-22T01:10:30.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.719+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f0108ac0 con 0x7f22f01017a0 2024-04-22T01:10:30.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.720+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f22ec044070 con 0x7f22f01017a0 2024-04-22T01:10:30.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.721+0000 7f22e6ffd640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22dc0a2c10 0x7f22dc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:30.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.721+0000 7f22f57e6640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22dc0a2c10 0x7f22dc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:30.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.721+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f22ec0f0c80 con 0x7f22f01017a0 2024-04-22T01:10:30.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.721+0000 7f22f57e6640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22dc0a2c10 0x7f22dc0a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f22f0198f90 tx=0x7f22e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:30.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.724+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f22ec0bdd20 con 0x7f22f01017a0 2024-04-22T01:10:30.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2739992022' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-22T01:10:30.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T01:10:30.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/730857694' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:10:30.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:30 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2739992022' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/730857694' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:30.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:10:30.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:30 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:30.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:30.918+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f22f002d3c0 con 0x7f22f01017a0 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2739992022' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/730857694' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:10:31.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:10:31.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:30 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.446+0000 7f22e6ffd640 1 -- 172.21.15.38:0/3247468289 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f22ec0bd620 con 0x7f22f01017a0 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22dc0a2c10 msgr2=0x7f22dc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22dc0a2c10 0x7f22dc0a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f22f0198f90 tx=0x7f22e0005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 msgr2=0x7f22f0197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0197a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22ec000c00 tx=0x7f22ec02c640 comp rx=0 tx=0).stop 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 shutdown_connections 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22dc0a2c10 0x7f22dc0a50d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f0103350 0x7f22f019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22f0102990 0x7f22f0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 --2- 172.21.15.38:0/3247468289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22f01017a0 0x7f22f0197a80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.448+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 >> 172.21.15.38:0/3247468289 conn(0x7f22f00fcd30 msgr2=0x7f22f0105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:31.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.449+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 shutdown_connections 2024-04-22T01:10:31.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.449+0000 7f22f824b640 1 -- 172.21.15.38:0/3247468289 wait complete. 2024-04-22T01:10:31.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 -- 172.21.15.38:0/1608159514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4988103360 msgr2=0x7f4988109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 --2- 172.21.15.38:0/1608159514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4988103360 0x7f4988109bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4980009ca0 tx=0x7f498002c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 -- 172.21.15.38:0/1608159514 shutdown_connections 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 --2- 172.21.15.38:0/1608159514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4988103360 0x7f4988109bf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 --2- 172.21.15.38:0/1608159514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49881029a0 0x7f4988102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 --2- 172.21.15.38:0/1608159514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 0x7f4988101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.578+0000 7f498cd1e640 1 -- 172.21.15.38:0/1608159514 >> 172.21.15.38:0/1608159514 conn(0x7f49880fcd40 msgr2=0x7f49880ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 -- 172.21.15.38:0/1608159514 shutdown_connections 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 -- 172.21.15.38:0/1608159514 wait complete. 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 Processor -- start 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 -- start start 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 0x7f49881979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 0x7f4988197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4988103360 0x7f498819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f498810d430 con 0x7f49881029a0 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f498810d2b0 con 0x7f49881017b0 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.579+0000 7f498cd1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f498810d5b0 con 0x7f4988103360 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 0x7f4988197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 0x7f4988197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38910/0 (socket says 172.21.15.38:38910) 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 -- 172.21.15.38:0/848842836 learned_addr learned my addr 172.21.15.38:0/848842836 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:31.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f498659c640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 0x7f49881979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f4986d9d640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4988103360 0x7f498819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 -- 172.21.15.38:0/848842836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4988103360 msgr2=0x7f498819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4988103360 0x7f498819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 -- 172.21.15.38:0/848842836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 msgr2=0x7f49881979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 0x7f49881979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 -- 172.21.15.38:0/848842836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498819f5d0 con 0x7f49881029a0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f498659c640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 0x7f49881979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497ffff640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 0x7f4988197ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f497000d770 tx=0x7f497000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970017070 con 0x7f49881029a0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498819f8c0 con 0x7f49881029a0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4970005500 con 0x7f49881029a0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970004e00 con 0x7f49881029a0 2024-04-22T01:10:31.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.580+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498819fdc0 con 0x7f49881029a0 2024-04-22T01:10:31.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.581+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4988108a00 con 0x7f49881029a0 2024-04-22T01:10:31.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.582+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4970004fa0 con 0x7f49881029a0 2024-04-22T01:10:31.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.583+0000 7f497f7fe640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f49740a2b90 0x7f49740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:31.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.583+0000 7f498659c640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f49740a2b90 0x7f49740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:31.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.583+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f49700ceba0 con 0x7f49881029a0 2024-04-22T01:10:31.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.583+0000 7f498659c640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f49740a2b90 0x7f49740a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4988102800 tx=0x7f497800a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:31.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.587+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f497009bad0 con 0x7f49881029a0 2024-04-22T01:10:31.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:31.775+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4988062340 con 0x7f49881029a0 2024-04-22T01:10:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: pgmap v290: 97 pgs: 16 creating+peering, 1 active+clean, 80 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T01:10:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/730857694' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3247468289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3247468289' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-mon[31057]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T01:10:31.896 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:10:31 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b[61609]: 2024-04-22T01:10:31.617+0000 7f27276bc700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: pgmap v290: 97 pgs: 16 creating+peering, 1 active+clean, 80 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/730857694' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3247468289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3247468289' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-22T01:10:31.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:31 smithi038 ceph-mon[28043]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T01:10:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: pgmap v290: 97 pgs: 16 creating+peering, 1 active+clean, 80 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:10:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T01:10:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/730857694' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T01:10:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T01:10:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T01:10:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:10:32.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3247468289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-22T01:10:32.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3247468289' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-22T01:10:32.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:31 smithi203 ceph-mon[31825]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T01:10:32.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:32 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] up:active 2024-04-22T01:10:32.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:32 smithi111 ceph-mon[31057]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:32.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:32 smithi111 ceph-mon[31057]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:32.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:32 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/848842836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T01:10:32.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:32 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] up:active 2024-04-22T01:10:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:32 smithi038 ceph-mon[28043]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:32 smithi038 ceph-mon[28043]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:32.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:32 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/848842836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T01:10:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:32 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] up:active 2024-04-22T01:10:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:32 smithi203 ceph-mon[31825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:32 smithi203 ceph-mon[31825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:33.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:32 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/848842836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T01:10:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.625+0000 7f497f7fe640 1 -- 172.21.15.38:0/848842836 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f497009b3d0 con 0x7f49881029a0 2024-04-22T01:10:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data_ec' created 2024-04-22T01:10:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f49740a2b90 msgr2=0x7f49740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f49740a2b90 0x7f49740a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4988102800 tx=0x7f497800a5e0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 msgr2=0x7f4988197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 0x7f4988197ef0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f497000d770 tx=0x7f497000dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 shutdown_connections 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f49740a2b90 0x7f49740a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4988103360 0x7f498819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49881029a0 0x7f4988197ef0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 --2- 172.21.15.38:0/848842836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881017b0 0x7f49881979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 >> 172.21.15.38:0/848842836 conn(0x7f49880fcd40 msgr2=0x7f4988105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 shutdown_connections 2024-04-22T01:10:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.628+0000 7f498cd1e640 1 -- 172.21.15.38:0/848842836 wait complete. 2024-04-22T01:10:33.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:33 smithi038 ceph-mon[28043]: pgmap v293: 97 pgs: 27 creating+peering, 34 active+clean, 36 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-22T01:10:33.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:33 smithi038 ceph-mon[28043]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T01:10:33.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:33 smithi038 ceph-mon[28043]: Cluster is now healthy 2024-04-22T01:10:33.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:33 smithi038 ceph-mon[28043]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T01:10:33.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-22T01:10:33.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 -- 172.21.15.38:0/2664496971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 msgr2=0x7f9820102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 --2- 172.21.15.38:0/2664496971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820102e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9814009900 tx=0x7f981402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 -- 172.21.15.38:0/2664496971 shutdown_connections 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 --2- 172.21.15.38:0/2664496971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 0x7f9820109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 --2- 172.21.15.38:0/2664496971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820102e10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 --2- 172.21.15.38:0/2664496971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98201017a0 0x7f9820101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 -- 172.21.15.38:0/2664496971 >> 172.21.15.38:0/2664496971 conn(0x7f98200fcd10 msgr2=0x7f98200ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 -- 172.21.15.38:0/2664496971 shutdown_connections 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.752+0000 7f982693e640 1 -- 172.21.15.38:0/2664496971 wait complete. 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 Processor -- start 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 -- start start 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98201017a0 0x7f98200754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 0x7f9820074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f982010d400 con 0x7f9820102990 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f982010d280 con 0x7f98201017a0 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f982693e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f982010d580 con 0x7f9820103350 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38942/0 (socket says 172.21.15.38:38942) 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f981f7fe640 1 -- 172.21.15.38:0/1560175158 learned_addr learned my addr 172.21.15.38:0/1560175158 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.753+0000 7f9824edb640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 0x7f9820074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981ffff640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98201017a0 0x7f98200754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981f7fe640 1 -- 172.21.15.38:0/1560175158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 msgr2=0x7f9820074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981f7fe640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 0x7f9820074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981f7fe640 1 -- 172.21.15.38:0/1560175158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98201017a0 msgr2=0x7f98200754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981f7fe640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98201017a0 0x7f98200754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981f7fe640 1 -- 172.21.15.38:0/1560175158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9820074900 con 0x7f9820102990 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f9824edb640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 0x7f9820074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:33.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981f7fe640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820073b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9814009230 tx=0x7f9814005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:33.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814041070 con 0x7f9820102990 2024-04-22T01:10:33.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98140054e0 con 0x7f9820102990 2024-04-22T01:10:33.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98201a1b70 con 0x7f9820102990 2024-04-22T01:10:33.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814034440 con 0x7f9820102990 2024-04-22T01:10:33.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.754+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98201a2110 con 0x7f9820102990 2024-04-22T01:10:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.755+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9820108b40 con 0x7f9820102990 2024-04-22T01:10:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.756+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9814044070 con 0x7f9820102990 2024-04-22T01:10:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.757+0000 7f981d7fa640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98080a2c10 0x7f98080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.757+0000 7f981ffff640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98080a2c10 0x7f98080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.757+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f98140f0fe0 con 0x7f9820102990 2024-04-22T01:10:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.758+0000 7f981ffff640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98080a2c10 0x7f98080a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9810005fd0 tx=0x7f9810005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:33.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.762+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f98140bdcc0 con 0x7f9820102990 2024-04-22T01:10:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:33 smithi111 ceph-mon[31057]: pgmap v293: 97 pgs: 27 creating+peering, 34 active+clean, 36 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-22T01:10:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:33 smithi111 ceph-mon[31057]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T01:10:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:33 smithi111 ceph-mon[31057]: Cluster is now healthy 2024-04-22T01:10:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:33 smithi111 ceph-mon[31057]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T01:10:33.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:33.948+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f982002d170 con 0x7f9820102990 2024-04-22T01:10:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:33 smithi203 ceph-mon[31825]: pgmap v293: 97 pgs: 27 creating+peering, 34 active+clean, 36 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-22T01:10:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:33 smithi203 ceph-mon[31825]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T01:10:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:33 smithi203 ceph-mon[31825]: Cluster is now healthy 2024-04-22T01:10:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:33 smithi203 ceph-mon[31825]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T01:10:34.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.635+0000 7f981d7fa640 1 -- 172.21.15.38:0/1560175158 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f98140bd5c0 con 0x7f9820102990 2024-04-22T01:10:34.636 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 allow_ec_overwrites to true 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.638+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98080a2c10 msgr2=0x7f98080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.638+0000 7f982693e640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98080a2c10 0x7f98080a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9810005fd0 tx=0x7f9810005c70 comp rx=0 tx=0).stop 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.638+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 msgr2=0x7f9820073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.638+0000 7f982693e640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820073b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9814009230 tx=0x7f9814005dd0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 shutdown_connections 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98080a2c10 0x7f98080a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9820103350 0x7f9820074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9820102990 0x7f9820073b30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 --2- 172.21.15.38:0/1560175158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98201017a0 0x7f98200754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 >> 172.21.15.38:0/1560175158 conn(0x7f98200fcd10 msgr2=0x7f9820105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:34.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 shutdown_connections 2024-04-22T01:10:34.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.639+0000 7f982693e640 1 -- 172.21.15.38:0/1560175158 wait complete. 2024-04-22T01:10:34.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 -- 172.21.15.38:0/11881019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d44102250 msgr2=0x7f7d44108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 --2- 172.21.15.38:0/11881019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d44102250 0x7f7d44108ae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009ca0 tx=0x7f7d3c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 -- 172.21.15.38:0/11881019 shutdown_connections 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 --2- 172.21.15.38:0/11881019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d44102250 0x7f7d44108ae0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 --2- 172.21.15.38:0/11881019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 0x7f7d44101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 --2- 172.21.15.38:0/11881019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d441006a0 0x7f7d44100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 -- 172.21.15.38:0/11881019 >> 172.21.15.38:0/11881019 conn(0x7f7d440fbc30 msgr2=0x7f7d440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:34.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 -- 172.21.15.38:0/11881019 shutdown_connections 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.774+0000 7f7d4893a640 1 -- 172.21.15.38:0/11881019 wait complete. 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 Processor -- start 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 -- start start 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 0x7f7d441969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 0x7f7d44196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d44102250 0x7f7d4419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4410c380 con 0x7f7d441006a0 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4410c200 con 0x7f7d44102250 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4893a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4410c500 con 0x7f7d44101890 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 0x7f7d441969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:34.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 0x7f7d441969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38962/0 (socket says 172.21.15.38:38962) 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 -- 172.21.15.38:0/3555950271 learned_addr learned my addr 172.21.15.38:0/3555950271 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d41d9b640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 0x7f7d44196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 -- 172.21.15.38:0/3555950271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 msgr2=0x7f7d44196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 0x7f7d44196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 -- 172.21.15.38:0/3555950271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d44102250 msgr2=0x7f7d4419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d44102250 0x7f7d4419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d4259c640 1 -- 172.21.15.38:0/3555950271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4419e5c0 con 0x7f7d441006a0 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.775+0000 7f7d41d9b640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 0x7f7d44196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.776+0000 7f7d4259c640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 0x7f7d441969a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c970 tx=0x7f7d3400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.776+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34007c10 con 0x7f7d441006a0 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.776+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d34007db0 con 0x7f7d441006a0 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.776+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3400f940 con 0x7f7d441006a0 2024-04-22T01:10:34.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.776+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4419e8b0 con 0x7f7d441006a0 2024-04-22T01:10:34.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.776+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4419ee00 con 0x7f7d441006a0 2024-04-22T01:10:34.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.777+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d44107960 con 0x7f7d441006a0 2024-04-22T01:10:34.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.778+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7d3400fae0 con 0x7f7d441006a0 2024-04-22T01:10:34.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.778+0000 7f7d337fe640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d2c0a2b90 0x7f7d2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:34.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.778+0000 7f7d41d9b640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d2c0a2b90 0x7f7d2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:34.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.779+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f7d340cf6f0 con 0x7f7d441006a0 2024-04-22T01:10:34.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.779+0000 7f7d41d9b640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d2c0a2b90 0x7f7d2c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7d38006fd0 tx=0x7f7d38008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:34.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:34.783+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7d3409c4c0 con 0x7f7d441006a0 2024-04-22T01:10:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:34 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/848842836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T01:10:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:34 smithi111 ceph-mon[31057]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T01:10:34.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:34 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1560175158' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-22T01:10:34.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:34 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/848842836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T01:10:34.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:34 smithi038 ceph-mon[28043]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T01:10:34.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:34 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1560175158' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-22T01:10:35.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.014+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f7d4402d420 con 0x7f7d441006a0 2024-04-22T01:10:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:34 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/848842836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T01:10:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:34 smithi203 ceph-mon[31825]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T01:10:35.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:34 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1560175158' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-22T01:10:35.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.646+0000 7f7d337fe640 1 -- 172.21.15.38:0/3555950271 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f7d3409bdc0 con 0x7f7d441006a0 2024-04-22T01:10:35.646 INFO:teuthology.orchestra.run.smithi038.stderr:added data pool 4 to fsmap 2024-04-22T01:10:35.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d2c0a2b90 msgr2=0x7f7d2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:35.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d2c0a2b90 0x7f7d2c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7d38006fd0 tx=0x7f7d38008040 comp rx=0 tx=0).stop 2024-04-22T01:10:35.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 msgr2=0x7f7d441969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:35.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 0x7f7d441969a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c970 tx=0x7f7d3400ce30 comp rx=0 tx=0).stop 2024-04-22T01:10:35.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 shutdown_connections 2024-04-22T01:10:35.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d2c0a2b90 0x7f7d2c0a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d44102250 0x7f7d4419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d44101890 0x7f7d44196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 --2- 172.21.15.38:0/3555950271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d441006a0 0x7f7d441969a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 >> 172.21.15.38:0/3555950271 conn(0x7f7d440fbc30 msgr2=0x7f7d44104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:35.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 shutdown_connections 2024-04-22T01:10:35.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.650+0000 7f7d4893a640 1 -- 172.21.15.38:0/3555950271 wait complete. 2024-04-22T01:10:35.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 -- 172.21.15.38:0/1169925341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4104ba0 msgr2=0x7f21f4106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 --2- 172.21.15.38:0/1169925341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4104ba0 0x7f21f4106f90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009900 tx=0x7f21dc02bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 -- 172.21.15.38:0/1169925341 shutdown_connections 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 --2- 172.21.15.38:0/1169925341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 0x7f21f41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 --2- 172.21.15.38:0/1169925341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4104ba0 0x7f21f4106f90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 --2- 172.21.15.38:0/1169925341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4102270 0x7f21f4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 -- 172.21.15.38:0/1169925341 >> 172.21.15.38:0/1169925341 conn(0x7f21f40fbc20 msgr2=0x7f21f40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 -- 172.21.15.38:0/1169925341 shutdown_connections 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.772+0000 7f21faea4640 1 -- 172.21.15.38:0/1169925341 wait complete. 2024-04-22T01:10:35.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 Processor -- start 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 -- start start 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 0x7f21f4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4104ba0 0x7f21f4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 0x7f21f419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410d460 con 0x7f21f4102270 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410d2e0 con 0x7f21f4104ba0 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21faea4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410d5e0 con 0x7f21f41074d0 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 0x7f21f4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 0x7f21f4197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38978/0 (socket says 172.21.15.38:38978) 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 -- 172.21.15.38:0/1019697819 learned_addr learned my addr 172.21.15.38:0/1019697819 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f9441640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 0x7f21f419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:35.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f3fff640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4104ba0 0x7f21f4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 -- 172.21.15.38:0/1019697819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 msgr2=0x7f21f419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 0x7f21f419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 -- 172.21.15.38:0/1019697819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4104ba0 msgr2=0x7f21f4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4104ba0 0x7f21f4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f8c40640 1 -- 172.21.15.38:0/1019697819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f419f640 con 0x7f21f4102270 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f9441640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 0x7f21f419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.773+0000 7f21f3fff640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4104ba0 0x7f21f4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.774+0000 7f21f8c40640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 0x7f21f4197a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f21e400c920 tx=0x7f21e400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.774+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007c70 con 0x7f21f4102270 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.774+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e4007e10 con 0x7f21f4102270 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.774+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f419f930 con 0x7f21f4102270 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.774+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4018890 con 0x7f21f4102270 2024-04-22T01:10:35.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.774+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f419fe80 con 0x7f21f4102270 2024-04-22T01:10:35.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.775+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f40fd710 con 0x7f21f4102270 2024-04-22T01:10:35.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.776+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f21e4018a70 con 0x7f21f4102270 2024-04-22T01:10:35.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.777+0000 7f21f1ffb640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21e00a2b90 0x7f21e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:35.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.777+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f21e40cf4e0 con 0x7f21f4102270 2024-04-22T01:10:35.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.777+0000 7f21f3fff640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21e00a2b90 0x7f21e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:35.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.777+0000 7f21f3fff640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21e00a2b90 0x7f21e00a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0096c0 tx=0x7f21dc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:35.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:35.781+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f21e4014030 con 0x7f21f4102270 2024-04-22T01:10:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:35 smithi111 ceph-mon[31057]: pgmap v296: 129 pgs: 22 creating+peering, 72 active+clean, 35 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-04-22T01:10:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:35 smithi111 ceph-mon[31057]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T01:10:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:35 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1560175158' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-22T01:10:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:35 smithi111 ceph-mon[31057]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T01:10:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:35 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3555950271' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-22T01:10:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:35 smithi111 ceph-mon[31057]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T01:10:35.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:35 smithi038 ceph-mon[28043]: pgmap v296: 129 pgs: 22 creating+peering, 72 active+clean, 35 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-04-22T01:10:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:35 smithi038 ceph-mon[28043]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T01:10:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:35 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1560175158' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-22T01:10:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:35 smithi038 ceph-mon[28043]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T01:10:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:35 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3555950271' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-22T01:10:35.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:35 smithi038 ceph-mon[28043]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T01:10:36.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.005+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21f402d170 con 0x7f21f4102270 2024-04-22T01:10:36.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.006+0000 7f21f1ffb640 1 -- 172.21.15.38:0/1019697819 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11812 (secure 0 0 0) 0x7f21e400de20 con 0x7f21f4102270 2024-04-22T01:10:36.006 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:36.007 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14595,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:10:35.016747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-04-22T01:10:36.007 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 17 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21e00a2b90 msgr2=0x7f21e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21e00a2b90 0x7f21e00a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0096c0 tx=0x7f21dc02df90 comp rx=0 tx=0).stop 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 msgr2=0x7f21f4197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 0x7f21f4197a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f21e400c920 tx=0x7f21e400cde0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 shutdown_connections 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21e00a2b90 0x7f21e00a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f41074d0 0x7f21f419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21f4104ba0 0x7f21f4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 --2- 172.21.15.38:0/1019697819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f21f4102270 0x7f21f4197a20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 >> 172.21.15.38:0/1019697819 conn(0x7f21f40fbc20 msgr2=0x7f21f4102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 shutdown_connections 2024-04-22T01:10:36.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.008+0000 7f21faea4640 1 -- 172.21.15.38:0/1019697819 wait complete. 2024-04-22T01:10:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:35 smithi203 ceph-mon[31825]: pgmap v296: 129 pgs: 22 creating+peering, 72 active+clean, 35 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-04-22T01:10:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:35 smithi203 ceph-mon[31825]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T01:10:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:35 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1560175158' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-22T01:10:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:35 smithi203 ceph-mon[31825]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T01:10:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:35 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3555950271' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-22T01:10:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:35 smithi203 ceph-mon[31825]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T01:10:36.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:36.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 -- 172.21.15.38:0/1560945272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 msgr2=0x7f7cc0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 --2- 172.21.15.38:0/1560945272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc0108ae0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009ca0 tx=0x7f7cb802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 -- 172.21.15.38:0/1560945272 shutdown_connections 2024-04-22T01:10:36.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 --2- 172.21.15.38:0/1560945272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc0108ae0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 --2- 172.21.15.38:0/1560945272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc0101890 0x7f7cc0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 --2- 172.21.15.38:0/1560945272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc01006a0 0x7f7cc0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 -- 172.21.15.38:0/1560945272 >> 172.21.15.38:0/1560945272 conn(0x7f7cc00fbc30 msgr2=0x7f7cc00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 -- 172.21.15.38:0/1560945272 shutdown_connections 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 -- 172.21.15.38:0/1560945272 wait complete. 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 Processor -- start 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.122+0000 7f7cc855b640 1 -- start start 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc01006a0 0x7f7cc006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc0101890 0x7f7cc019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc010c300 con 0x7f7cc0102250 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc010c180 con 0x7f7cc0101890 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc010c480 con 0x7f7cc01006a0 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc019ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39000/0 (socket says 172.21.15.38:39000) 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 -- 172.21.15.38:0/3580205574 learned_addr learned my addr 172.21.15.38:0/3580205574 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:36.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc5af6640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc0101890 0x7f7cc019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc62f7640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc01006a0 0x7f7cc006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 -- 172.21.15.38:0/3580205574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc01006a0 msgr2=0x7f7cc006a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc01006a0 0x7f7cc006a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 -- 172.21.15.38:0/3580205574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc0101890 msgr2=0x7f7cc019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc0101890 0x7f7cc019c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 -- 172.21.15.38:0/3580205574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc019f6a0 con 0x7f7cc0102250 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc6af8640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc019ede0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8037040 tx=0x7f7cb8032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8039040 con 0x7f7cc0102250 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cb8044070 con 0x7f7cc0102250 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc019f930 con 0x7f7cc0102250 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8004330 con 0x7f7cc0102250 2024-04-22T01:10:36.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.123+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc019fe20 con 0x7f7cc0102250 2024-04-22T01:10:36.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.124+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc0107a90 con 0x7f7cc0102250 2024-04-22T01:10:36.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.126+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7cb8003a00 con 0x7f7cc0102250 2024-04-22T01:10:36.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.126+0000 7f7cb77fe640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7cac0a2b90 0x7f7cac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.126+0000 7f7cc62f7640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7cac0a2b90 0x7f7cac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.126+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7cb80f20a0 con 0x7f7cc0102250 2024-04-22T01:10:36.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.127+0000 7f7cc62f7640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7cac0a2b90 0x7f7cac0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005db0 tx=0x7f7cbc005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:36.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.130+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7cb80beee0 con 0x7f7cc0102250 2024-04-22T01:10:36.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.315+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cc002d340 con 0x7f7cc0102250 2024-04-22T01:10:36.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.316+0000 7f7cb77fe640 1 -- 172.21.15.38:0/3580205574 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f7cb80efe20 con 0x7f7cc0102250 2024-04-22T01:10:36.317 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:36.317 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7cac0a2b90 msgr2=0x7f7cac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7cac0a2b90 0x7f7cac0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005db0 tx=0x7f7cbc005d40 comp rx=0 tx=0).stop 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 msgr2=0x7f7cc019ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc019ede0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8037040 tx=0x7f7cb8032f90 comp rx=0 tx=0).stop 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 shutdown_connections 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7cac0a2b90 0x7f7cac0a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cc0102250 0x7f7cc019ede0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc0101890 0x7f7cc019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 --2- 172.21.15.38:0/3580205574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc01006a0 0x7f7cc006a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 >> 172.21.15.38:0/3580205574 conn(0x7f7cc00fbc30 msgr2=0x7f7cc0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:36.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.318+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 shutdown_connections 2024-04-22T01:10:36.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.319+0000 7f7cc855b640 1 -- 172.21.15.38:0/3580205574 wait complete. 2024-04-22T01:10:36.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:36.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 -- 172.21.15.38:0/1528983181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c1074e0 msgr2=0x7f769c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1528983181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c1074e0 0x7f769c1098d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7694009ca0 tx=0x7f769402c100 comp rx=0 tx=0).stop 2024-04-22T01:10:36.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 -- 172.21.15.38:0/1528983181 shutdown_connections 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1528983181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c1074e0 0x7f769c1098d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1528983181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c104bb0 0x7f769c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1528983181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f769c102280 0x7f769c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 -- 172.21.15.38:0/1528983181 >> 172.21.15.38:0/1528983181 conn(0x7f769c0fbc30 msgr2=0x7f769c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 -- 172.21.15.38:0/1528983181 shutdown_connections 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.432+0000 7f76a42c6640 1 -- 172.21.15.38:0/1528983181 wait complete. 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 Processor -- start 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 -- start start 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c102280 0x7f769c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 0x7f769c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f769c1074e0 0x7f769c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f769c10d420 con 0x7f769c104bb0 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f769c10d2a0 con 0x7f769c102280 2024-04-22T01:10:36.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a42c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f769c10d5a0 con 0x7f769c1074e0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 0x7f769c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 0x7f769c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39014/0 (socket says 172.21.15.38:39014) 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 -- 172.21.15.38:0/1286517013 learned_addr learned my addr 172.21.15.38:0/1286517013 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a2863640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f769c1074e0 0x7f769c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a2062640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c102280 0x7f769c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 -- 172.21.15.38:0/1286517013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f769c1074e0 msgr2=0x7f769c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f769c1074e0 0x7f769c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 -- 172.21.15.38:0/1286517013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c102280 msgr2=0x7f769c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c102280 0x7f769c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a1861640 1 -- 172.21.15.38:0/1286517013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769c19f5f0 con 0x7f769c104bb0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.433+0000 7f76a2062640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c102280 0x7f769c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.434+0000 7f76a1861640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 0x7f769c197f10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f768c00c970 tx=0x7f768c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.434+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c007bb0 con 0x7f769c104bb0 2024-04-22T01:10:36.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.434+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f768c007d50 con 0x7f769c104bb0 2024-04-22T01:10:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.434+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f769c19f8e0 con 0x7f769c104bb0 2024-04-22T01:10:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.434+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c00f450 con 0x7f769c104bb0 2024-04-22T01:10:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.434+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f769c19fe30 con 0x7f769c104bb0 2024-04-22T01:10:36.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.435+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f769c0fd720 con 0x7f769c104bb0 2024-04-22T01:10:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.436+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f768c0046c0 con 0x7f769c104bb0 2024-04-22T01:10:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.437+0000 7f76937fe640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f76880a2b90 0x7f76880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.437+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f768c0cf210 con 0x7f769c104bb0 2024-04-22T01:10:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.437+0000 7f76a2062640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f76880a2b90 0x7f76880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.437+0000 7f76a2062640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f76880a2b90 0x7f76880a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f769800abe0 tx=0x7f769800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:36.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.440+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f768c014030 con 0x7f769c104bb0 2024-04-22T01:10:36.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.628+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f769c02d170 con 0x7f769c104bb0 2024-04-22T01:10:36.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.630+0000 7f76937fe640 1 -- 172.21.15.38:0/1286517013 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f768c09bfd0 con 0x7f769c104bb0 2024-04-22T01:10:36.631 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:36.631 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:36.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f76880a2b90 msgr2=0x7f76880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f76880a2b90 0x7f76880a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f769800abe0 tx=0x7f769800a5c0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 msgr2=0x7f769c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 0x7f769c197f10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f768c00c970 tx=0x7f768c00ce30 comp rx=0 tx=0).stop 2024-04-22T01:10:36.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 shutdown_connections 2024-04-22T01:10:36.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f76880a2b90 0x7f76880a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f769c1074e0 0x7f769c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f769c104bb0 0x7f769c197f10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 --2- 172.21.15.38:0/1286517013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f769c102280 0x7f769c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 >> 172.21.15.38:0/1286517013 conn(0x7f769c0fbc30 msgr2=0x7f769c102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:36.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 shutdown_connections 2024-04-22T01:10:36.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.632+0000 7f76a42c6640 1 -- 172.21.15.38:0/1286517013 wait complete. 2024-04-22T01:10:36.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 -- 172.21.15.38:0/3443433949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb420102250 msgr2=0x7fb420108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 --2- 172.21.15.38:0/3443433949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb420102250 0x7fb420108ae0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb418009ca0 tx=0x7fb41802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 -- 172.21.15.38:0/3443433949 shutdown_connections 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 --2- 172.21.15.38:0/3443433949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb420102250 0x7fb420108ae0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 --2- 172.21.15.38:0/3443433949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 0x7fb420101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 --2- 172.21.15.38:0/3443433949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4201006a0 0x7fb420100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 -- 172.21.15.38:0/3443433949 >> 172.21.15.38:0/3443433949 conn(0x7fb4200fbc30 msgr2=0x7fb4200fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 -- 172.21.15.38:0/3443433949 shutdown_connections 2024-04-22T01:10:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.762+0000 7fb42771e640 1 -- 172.21.15.38:0/3443433949 wait complete. 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 Processor -- start 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 -- start start 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 0x7fb42006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 0x7fb42019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb420102250 0x7fb42019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010c380 con 0x7fb4201006a0 2024-04-22T01:10:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010c200 con 0x7fb420102250 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb42771e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010c500 con 0x7fb420101890 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 0x7fb42006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 0x7fb42006a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49262/0 (socket says 172.21.15.38:49262) 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 -- 172.21.15.38:0/3058439236 learned_addr learned my addr 172.21.15.38:0/3058439236 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 -- 172.21.15.38:0/3058439236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 msgr2=0x7fb42019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb424cb9640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 0x7fb42019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 0x7fb42019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 -- 172.21.15.38:0/3058439236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb420102250 msgr2=0x7fb42019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.763+0000 7fb4254ba640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb420102250 0x7fb42019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb4254ba640 1 -- 172.21.15.38:0/3058439236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42019f660 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb424cb9640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 0x7fb42019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb4254ba640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 0x7fb42006a910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c970 tx=0x7fb41000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410007c10 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb410007db0 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42019f8f0 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41000f940 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.764+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42019fe40 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.765+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb420107960 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.766+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb41000fae0 con 0x7fb4201006a0 2024-04-22T01:10:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.766+0000 7fb4167fc640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb40c0a2b90 0x7fb40c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.767+0000 7fb424cb9640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb40c0a2b90 0x7fb40c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.767+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb4100cf790 con 0x7fb4201006a0 2024-04-22T01:10:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.767+0000 7fb424cb9640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb40c0a2b90 0x7fb40c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006fd0 tx=0x7fb41c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.771+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb41009c4e0 con 0x7fb4201006a0 2024-04-22T01:10:36.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3555950271' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-22T01:10:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:36 smithi038 ceph-mon[28043]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1019697819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3580205574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:36.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1286517013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:36.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.996+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb42002d170 con 0x7fb4201006a0 2024-04-22T01:10:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.997+0000 7fb4167fc640 1 -- 172.21.15.38:0/3058439236 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11812 (secure 0 0 0) 0x7fb42002d170 con 0x7fb4201006a0 2024-04-22T01:10:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:36.998 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14595,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24400,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24439,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:10:35.016747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-04-22T01:10:36.998 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 17 2024-04-22T01:10:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb40c0a2b90 msgr2=0x7fb40c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb40c0a2b90 0x7fb40c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006fd0 tx=0x7fb41c008040 comp rx=0 tx=0).stop 2024-04-22T01:10:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 msgr2=0x7fb42006a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 0x7fb42006a910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c970 tx=0x7fb41000ce30 comp rx=0 tx=0).stop 2024-04-22T01:10:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 shutdown_connections 2024-04-22T01:10:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb40c0a2b90 0x7fb40c0a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb420102250 0x7fb42019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb420101890 0x7fb42019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 --2- 172.21.15.38:0/3058439236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4201006a0 0x7fb42006a910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 >> 172.21.15.38:0/3058439236 conn(0x7fb4200fbc30 msgr2=0x7fb420104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 shutdown_connections 2024-04-22T01:10:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:36.999+0000 7fb42771e640 1 -- 172.21.15.38:0/3058439236 wait complete. 2024-04-22T01:10:37.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3555950271' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-22T01:10:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:36 smithi203 ceph-mon[31825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1019697819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3580205574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1286517013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 -- 172.21.15.38:0/1879839316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c102250 msgr2=0x7f4d6c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 --2- 172.21.15.38:0/1879839316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c102250 0x7f4d6c108ae0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009ca0 tx=0x7f4d6402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 -- 172.21.15.38:0/1879839316 shutdown_connections 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 --2- 172.21.15.38:0/1879839316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c102250 0x7f4d6c108ae0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 --2- 172.21.15.38:0/1879839316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 0x7f4d6c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 --2- 172.21.15.38:0/1879839316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 -- 172.21.15.38:0/1879839316 >> 172.21.15.38:0/1879839316 conn(0x7f4d6c0fbc30 msgr2=0x7f4d6c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 -- 172.21.15.38:0/1879839316 shutdown_connections 2024-04-22T01:10:37.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 -- 172.21.15.38:0/1879839316 wait complete. 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 Processor -- start 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.114+0000 7f4d7264e640 1 -- start start 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 0x7f4d6c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d6c102250 0x7f4d6c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c10c300 con 0x7f4d6c1006a0 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c10c180 con 0x7f4d6c101890 2024-04-22T01:10:37.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c10c480 con 0x7f4d6c102250 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49276/0 (socket says 172.21.15.38:49276) 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 -- 172.21.15.38:0/249519092 learned_addr learned my addr 172.21.15.38:0/249519092 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6b7fe640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 0x7f4d6c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 -- 172.21.15.38:0/249519092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d6c102250 msgr2=0x7f4d6c1031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d6c102250 0x7f4d6c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 -- 172.21.15.38:0/249519092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 msgr2=0x7f4d6c102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 0x7f4d6c102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 -- 172.21.15.38:0/249519092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c103a40 con 0x7f4d6c1006a0 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6b7fe640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 0x7f4d6c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d6bfff640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c104630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00c970 tx=0x7f4d5c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c007c10 con 0x7f4d6c1006a0 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d5c007db0 con 0x7f4d6c1006a0 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6c1a1ab0 con 0x7f4d6c1006a0 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.115+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c00f940 con 0x7f4d6c1006a0 2024-04-22T01:10:37.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.116+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6c1a2000 con 0x7f4d6c1006a0 2024-04-22T01:10:37.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.116+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d6c1a21d0 con 0x7f4d6c1006a0 2024-04-22T01:10:37.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.117+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4d5c00fae0 con 0x7f4d6c1006a0 2024-04-22T01:10:37.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.118+0000 7f4d697fa640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4d540a2b90 0x7f4d540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.118+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4d5c0cf760 con 0x7f4d6c1006a0 2024-04-22T01:10:37.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.118+0000 7f4d6b7fe640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4d540a2b90 0x7f4d540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.119+0000 7f4d6b7fe640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4d540a2b90 0x7f4d540a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c1043a0 tx=0x7f4d60005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.123+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4d5c09c510 con 0x7f4d6c1006a0 2024-04-22T01:10:37.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3555950271' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-22T01:10:37.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:36 smithi111 ceph-mon[31057]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:37.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1019697819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3580205574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1286517013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.313+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d6c1a2ac0 con 0x7f4d6c1006a0 2024-04-22T01:10:37.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.314+0000 7f4d697fa640 1 -- 172.21.15.38:0/249519092 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f4d5c09be10 con 0x7f4d6c1006a0 2024-04-22T01:10:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:37.315 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4d540a2b90 msgr2=0x7f4d540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4d540a2b90 0x7f4d540a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c1043a0 tx=0x7f4d60005e50 comp rx=0 tx=0).stop 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 msgr2=0x7f4d6c104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c104630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00c970 tx=0x7f4d5c00ce30 comp rx=0 tx=0).stop 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 shutdown_connections 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4d540a2b90 0x7f4d540a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d6c102250 0x7f4d6c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d6c101890 0x7f4d6c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 --2- 172.21.15.38:0/249519092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d6c1006a0 0x7f4d6c104630 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 >> 172.21.15.38:0/249519092 conn(0x7f4d6c0fbc30 msgr2=0x7f4d6c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.316+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 shutdown_connections 2024-04-22T01:10:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.317+0000 7f4d7264e640 1 -- 172.21.15.38:0/249519092 wait complete. 2024-04-22T01:10:37.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-22T01:10:29.943110+0000', 'modified': '2024-04-22T01:10:35.016747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3001874105', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3001874105}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3001874105}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24527, 'qdb_cluster': [24527]} 2024-04-22T01:10:37.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-22T01:10:37.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 -- 172.21.15.38:0/3540907497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 msgr2=0x7fb67c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 --2- 172.21.15.38:0/3540907497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c108ae0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb674009ca0 tx=0x7fb67402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 -- 172.21.15.38:0/3540907497 shutdown_connections 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 --2- 172.21.15.38:0/3540907497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c108ae0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 --2- 172.21.15.38:0/3540907497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 0x7fb67c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 --2- 172.21.15.38:0/3540907497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb67c1006a0 0x7fb67c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 -- 172.21.15.38:0/3540907497 >> 172.21.15.38:0/3540907497 conn(0x7fb67c0fbc10 msgr2=0x7fb67c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 -- 172.21.15.38:0/3540907497 shutdown_connections 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.431+0000 7fb6808aa640 1 -- 172.21.15.38:0/3540907497 wait complete. 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 Processor -- start 2024-04-22T01:10:37.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 -- start start 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb67c1006a0 0x7fb67c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 0x7fb67c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10c300 con 0x7fb67c102250 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10c180 con 0x7fb67c1006a0 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb6808aa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10c480 con 0x7fb67c101890 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb67ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb67ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49296/0 (socket says 172.21.15.38:49296) 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb67ad9d640 1 -- 172.21.15.38:0/2386009738 learned_addr learned my addr 172.21.15.38:0/2386009738 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb67a59c640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb67c1006a0 0x7fb67c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.432+0000 7fb673fff640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 0x7fb67c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb67ad9d640 1 -- 172.21.15.38:0/2386009738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 msgr2=0x7fb67c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb67ad9d640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 0x7fb67c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb67ad9d640 1 -- 172.21.15.38:0/2386009738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb67c1006a0 msgr2=0x7fb67c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb67ad9d640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb67c1006a0 0x7fb67c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb67ad9d640 1 -- 172.21.15.38:0/2386009738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c19f650 con 0x7fb67c102250 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb673fff640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 0x7fb67c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb67ad9d640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c19efe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb674005e30 tx=0x7fb674032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb6737fe640 1 -- 172.21.15.38:0/2386009738 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674041020 con 0x7fb67c102250 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb6737fe640 1 -- 172.21.15.38:0/2386009738 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb674034cd0 con 0x7fb67c102250 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c19f8e0 con 0x7fb67c102250 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb6737fe640 1 -- 172.21.15.38:0/2386009738 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674039040 con 0x7fb67c102250 2024-04-22T01:10:37.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.433+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c19fe20 con 0x7fb67c102250 2024-04-22T01:10:37.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb6737fe640 1 -- 172.21.15.38:0/2386009738 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb674044020 con 0x7fb67c102250 2024-04-22T01:10:37.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 Processor -- start 2024-04-22T01:10:37.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 -- start start 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bac070 0x561709bac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709baca60 0x561709bb1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bb4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561709b10df0 con 0x561709bb1bd0 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561709ab9640 con 0x561709bac070 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb683568740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561709b0cdb0 con 0x561709baca60 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb673fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bb4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb673fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bb4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49298/0 (socket says 172.21.15.38:49298) 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.436+0000 7fb673fff640 1 -- 172.21.15.38:0/98504596 learned_addr learned my addr 172.21.15.38:0/98504596 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb67a59c640 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709baca60 0x561709bb1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bac070 0x561709bac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb6737fe640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb6680a2c10 0x7fb6680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 -- 172.21.15.38:0/98504596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709baca60 msgr2=0x561709bb1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709baca60 0x561709bb1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 -- 172.21.15.38:0/98504596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bac070 msgr2=0x561709bac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bac070 0x561709bac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 -- 172.21.15.38:0/98504596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617098252d0 con 0x561709bb1bd0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb67ad9d640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb6680a2c10 0x7fb6680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb6737fe640 1 -- 172.21.15.38:0/2386009738 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb6740f1200 con 0x7fb67c102250 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb673fff640 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bb4060 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b3c0 tx=0x7fb66400b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b97aea2586a2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb670ff9640 1 -- 172.21.15.38:0/98504596 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664004460 con 0x561709bb1bd0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb670ff9640 1 -- 172.21.15.38:0/98504596 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb664005500 con 0x561709bb1bd0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.437+0000 7fb670ff9640 1 -- 172.21.15.38:0/98504596 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664003850 con 0x561709bb1bd0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 -- 172.21.15.38:0/98504596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 msgr2=0x561709bb4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb67ad9d640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb6680a2c10 0x7fb6680a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb6740ba290 tx=0x7fb674088490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bb4060 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb66400b3c0 tx=0x7fb66400b6e0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 -- 172.21.15.38:0/98504596 shutdown_connections 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bb4060 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709baca60 0x561709bb1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 --2- 172.21.15.38:0/98504596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bac070 0x561709bac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 -- 172.21.15.38:0/98504596 >> 172.21.15.38:0/98504596 conn(0x561709ba75e0 msgr2=0x561709ba79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 -- 172.21.15.38:0/98504596 shutdown_connections 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 -- 172.21.15.38:0/98504596 wait complete. 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 -- start start 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.438+0000 7fb683568740 1 Processor -- start 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bb6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709bb6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bbdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561709bb4b80 con 0x561709bb1bd0 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561709accbe0 con 0x561709baca60 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561709bb4d00 con 0x561709bac070 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 10 client.0 _ll_register_callbacks cb 0x561709acc5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:10:37.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bbdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bbdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49308/0 (socket says 172.21.15.38:49308) 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 -- 172.21.15.38:0/615723068 learned_addr learned my addr 172.21.15.38:0/615723068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bb6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb67a59c640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709bb6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 -- 172.21.15.38:0/615723068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 msgr2=0x561709bb6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bb6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 -- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 msgr2=0x561709bb6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709bb6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 -- 172.21.15.38:0/615723068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617098ef6b0 con 0x561709bb1bd0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb67a59c640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709bb6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb673fff640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bbdb10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb66400bc40 tx=0x7fb66400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534128efe8ca015f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb661ffb640 1 -- 172.21.15.38:0/615723068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664025040 con 0x561709bb1bd0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb661ffb640 1 -- 172.21.15.38:0/615723068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb6640046c0 con 0x561709bb1bd0 2024-04-22T01:10:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.439+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb664005500 con 0x561709bb1bd0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb661ffb640 1 -- 172.21.15.38:0/615723068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fb664033050 con 0x561709bb1bd0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 10 client.14727 fetch_fsmap learned FSMap version 17 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561709bc0b20 con 0x561709bb1bd0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb661ffb640 1 -- 172.21.15.38:0/615723068 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb664005500 con 0x561709bb1bd0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb661ffb640 1 -- 172.21.15.38:0/615723068 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fb66401b470 con 0x561709bb1bd0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 10 client.14727 fetch_fsmap finished waiting for FSMap version 17 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 20 client.14727 populate_metadata read hostname 'smithi038' 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709bc2fb0 0x561709bc4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 4 client.14727 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561709b1e330 con 0x561709bc2fb0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.440+0000 7fb67a59c640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709bc2fb0 0x561709bc4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.441+0000 7fb67a59c640 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709bc2fb0 0x561709bc4420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6355f5bd339a6dc5 server_cookie=781f2f632e4560b9 in_seq=0 out_seq=0 2024-04-22T01:10:37.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.441+0000 7fb661ffb640 10 client.14727 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb661ffb640 1 -- 172.21.15.38:0/615723068 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561709b1e330 con 0x561709bc2fb0 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb661ffb640 10 client.14727 handle_command_reply: tid=0 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 client.14727 shutdown 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 msgr2=0x561709bbdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bbdb10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb66400bc40 tx=0x7fb66400b460 comp rx=0 tx=0).stop 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 shutdown_connections 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709bc2fb0 0x561709bc4420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bb1bd0 0x561709bbdb10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709bb6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb661ffb640 0 client.14727 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 --2- 172.21.15.38:0/615723068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bb6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 >> 172.21.15.38:0/615723068 conn(0x561709ba75e0 msgr2=0x561709ba6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 shutdown_connections 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 1 -- 172.21.15.38:0/615723068 wait complete. 2024-04-22T01:10:37.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.442+0000 7fb683568740 20 client.14727 trim_cache size 0 max 16384 2024-04-22T01:10:37.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 Processor -- start 2024-04-22T01:10:37.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 -- start start 2024-04-22T01:10:37.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 0x561709ba79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709badec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bb1bd0 0x561709bae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561709bc2760 con 0x561709bac070 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561709bc2e30 con 0x561709baca60 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb683568740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561709bc3500 con 0x561709bb1bd0 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb673fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 0x561709ba79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb673fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 0x561709ba79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49318/0 (socket says 172.21.15.38:49318) 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.460+0000 7fb673fff640 1 -- 172.21.15.38:0/1289735401 learned_addr learned my addr 172.21.15.38:0/1289735401 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb673fff640 1 -- 172.21.15.38:0/1289735401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bb1bd0 msgr2=0x561709bae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb673fff640 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bb1bd0 0x561709bae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb673fff640 1 -- 172.21.15.38:0/1289735401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 msgr2=0x561709badec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb673fff640 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709badec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb673fff640 1 -- 172.21.15.38:0/1289735401 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561709ba5c20 con 0x561709bac070 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb673fff640 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 0x561709ba79c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb66400c010 tx=0x7fb66401cfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87acc75ba1f7c9a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb663fff640 1 -- 172.21.15.38:0/1289735401 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664010cf0 con 0x561709bac070 2024-04-22T01:10:37.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb663fff640 1 -- 172.21.15.38:0/1289735401 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66401b470 con 0x561709bac070 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb663fff640 1 -- 172.21.15.38:0/1289735401 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6640048e0 con 0x561709bac070 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb683568740 1 -- 172.21.15.38:0/1289735401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 msgr2=0x561709ba79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb683568740 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 0x561709ba79c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb66400c010 tx=0x7fb66401cfe0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb683568740 1 -- 172.21.15.38:0/1289735401 shutdown_connections 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb683568740 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bb1bd0 0x561709bae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb683568740 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709baca60 0x561709badec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.461+0000 7fb683568740 1 --2- 172.21.15.38:0/1289735401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709bac070 0x561709ba79c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- 172.21.15.38:0/1289735401 >> 172.21.15.38:0/1289735401 conn(0x561709ba75e0 msgr2=0x561709bc1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- 172.21.15.38:0/1289735401 shutdown_connections 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- 172.21.15.38:0/1289735401 wait complete. 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- start start 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 Processor -- start 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bbda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 0x561709c28f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bb1bd0 0x561709c2b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561709bc42a0 con 0x561709baca60 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561709bc3bd0 con 0x561709bb1bd0 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561709bc4970 con 0x561709bac070 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb683568740 10 client.0 _ll_register_callbacks cb 0x7fb6640050b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 0x561709c28f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 0x561709c28f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49332/0 (socket says 172.21.15.38:49332) 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb67a59c640 1 -- 172.21.15.38:0/1948383647 learned_addr learned my addr 172.21.15.38:0/1948383647 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb673fff640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bbda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.462+0000 7fb673fff640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bb1bd0 0x561709c2b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb67a59c640 1 -- 172.21.15.38:0/1948383647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 msgr2=0x561709bbda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb67a59c640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bbda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb67a59c640 1 -- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bb1bd0 msgr2=0x561709c2b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:10:37.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb67a59c640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bb1bd0 0x561709c2b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb67a59c640 1 -- 172.21.15.38:0/1948383647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561709c2bcc0 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb673fff640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bb1bd0 0x561709c2b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb67a59c640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 0x561709c28f60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00bde0 tx=0x7fb66c00bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18aa302e37da13e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb661ffb640 1 -- 172.21.15.38:0/1948383647 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c01d470 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb661ffb640 1 -- 172.21.15.38:0/1948383647 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb66c01d610 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb66401b470 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb661ffb640 1 -- 172.21.15.38:0/1948383647 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fb66c02c050 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 10 client.14733 fetch_fsmap learned FSMap version 17 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561709c2d0a0 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb661ffb640 1 -- 172.21.15.38:0/1948383647 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66401b470 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb661ffb640 1 -- 172.21.15.38:0/1948383647 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fb66c010430 con 0x561709baca60 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 10 client.14733 fetch_fsmap finished waiting for FSMap version 17 2024-04-22T01:10:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 10 client.14733 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 20 client.14733 populate_metadata read hostname 'smithi038' 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709c31cd0 0x561709c309e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 4 client.14733 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.463+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "status"}) -- 0x561709b18680 con 0x561709c31cd0 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb673fff640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709c31cd0 0x561709c309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb661ffb640 10 client.14733 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb673fff640 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709c31cd0 0x561709c309e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d6194920d45122 server_cookie=353d351fb5e17529 in_seq=0 out_seq=0 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb661ffb640 1 -- 172.21.15.38:0/1948383647 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x561709b18680 con 0x561709c31cd0 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb661ffb640 10 client.14733 handle_command_reply: tid=0 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 client.14733 shutdown 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 msgr2=0x561709c28f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 0x561709c28f60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00bde0 tx=0x7fb66c00bc60 comp rx=0 tx=0).stop 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 shutdown_connections 2024-04-22T01:10:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561709c31cd0 0x561709c309e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561709bb1bd0 0x561709c2b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561709baca60 0x561709c28f60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 --2- 172.21.15.38:0/1948383647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561709bac070 0x561709bbda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb661ffb640 0 client.14733 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.464+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 >> 172.21.15.38:0/1948383647 conn(0x561709ba75e0 msgr2=0x561709badec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 shutdown_connections 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb683568740 1 -- 172.21.15.38:0/1948383647 wait complete. 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb683568740 20 client.14733 trim_cache size 0 max 16384 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-04-22T01:10:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24527, 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 6.8720008210000003, 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 17, 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 11.03128781 2024-04-22T01:10:37.472 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb6680a2c10 msgr2=0x7fb6680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb6680a2c10 0x7fb6680a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb6740ba290 tx=0x7fb674088490 comp rx=0 tx=0).stop 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 msgr2=0x7fb67c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c19efe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb674005e30 tx=0x7fb674032ec0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 shutdown_connections 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb6680a2c10 0x7fb6680a50d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67c102250 0x7fb67c19efe0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb67c101890 0x7fb67c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 --2- 172.21.15.38:0/2386009738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb67c1006a0 0x7fb67c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 >> 172.21.15.38:0/2386009738 conn(0x7fb67c0fbc10 msgr2=0x7fb67c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:37.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 shutdown_connections 2024-04-22T01:10:37.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:37.465+0000 7fb6808aa640 1 -- 172.21.15.38:0/2386009738 wait complete. 2024-04-22T01:10:37.476 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:10:37.476 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:37.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T01:10:37.725 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:10:37.985 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:37 smithi038 ceph-mon[28043]: pgmap v299: 129 pgs: 5 creating+peering, 107 active+clean, 17 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2024-04-22T01:10:37.986 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:37 smithi038 ceph-mon[28043]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T01:10:37.986 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:37 smithi038 ceph-mon[28043]: Cluster is now healthy 2024-04-22T01:10:37.986 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:37 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3058439236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:37.986 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:37 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/249519092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:38.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:37 smithi203 ceph-mon[31825]: pgmap v299: 129 pgs: 5 creating+peering, 107 active+clean, 17 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2024-04-22T01:10:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:37 smithi203 ceph-mon[31825]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T01:10:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:37 smithi203 ceph-mon[31825]: Cluster is now healthy 2024-04-22T01:10:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:37 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3058439236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:37 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/249519092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:37 smithi111 ceph-mon[31057]: pgmap v299: 129 pgs: 5 creating+peering, 107 active+clean, 17 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2024-04-22T01:10:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:37 smithi111 ceph-mon[31057]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T01:10:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:37 smithi111 ceph-mon[31057]: Cluster is now healthy 2024-04-22T01:10:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:37 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3058439236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:10:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:37 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/249519092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 -- 172.21.15.38:0/726541951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 msgr2=0x7f649c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 --2- 172.21.15.38:0/726541951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c10f160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f649400b0a0 tx=0x7f649400b3b0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 -- 172.21.15.38:0/726541951 shutdown_connections 2024-04-22T01:10:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 --2- 172.21.15.38:0/726541951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c10f160 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 --2- 172.21.15.38:0/726541951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f649c102510 0x7f649c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 --2- 172.21.15.38:0/726541951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f649c102ec0 0x7f649c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 -- 172.21.15.38:0/726541951 >> 172.21.15.38:0/726541951 conn(0x7f649c0793a0 msgr2=0x7f649c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 -- 172.21.15.38:0/726541951 shutdown_connections 2024-04-22T01:10:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.547+0000 7f64a124d700 1 -- 172.21.15.38:0/726541951 wait complete. 2024-04-22T01:10:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 Processor -- start 2024-04-22T01:10:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 -- start start 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f649c102510 0x7f649c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f649c102ec0 0x7f649c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f649c1125d0 con 0x7f649c107590 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f649c112450 con 0x7f649c102ec0 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f64a124d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f649c112750 con 0x7f649c102510 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f649b59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f649b59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49354/0 (socket says 172.21.15.38:49354) 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f649b59e700 1 -- 172.21.15.38:0/3568912750 learned_addr learned my addr 172.21.15.38:0/3568912750 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.548+0000 7f649ad9d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f649c102510 0x7f649c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649a59c700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f649c102ec0 0x7f649c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649b59e700 1 -- 172.21.15.38:0/3568912750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f649c102510 msgr2=0x7f649c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649b59e700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f649c102510 0x7f649c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649b59e700 1 -- 172.21.15.38:0/3568912750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f649c102ec0 msgr2=0x7f649c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649b59e700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f649c102ec0 0x7f649c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649b59e700 1 -- 172.21.15.38:0/3568912750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c1a6f60 con 0x7f649c107590 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f649b59e700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c074fb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f64940062a0 tx=0x7f649400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494021070 con 0x7f649c107590 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6494007c80 con 0x7f649c107590 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649c1a71f0 con 0x7f649c107590 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494005420 con 0x7f649c107590 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.549+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f649c1a76a0 con 0x7f649c107590 2024-04-22T01:10:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.550+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f649c068f20 con 0x7f649c107590 2024-04-22T01:10:39.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.551+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6494024070 con 0x7f649c107590 2024-04-22T01:10:39.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.552+0000 7f6493fff700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f64880a2c20 0x7f64880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:39.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.552+0000 7f649ad9d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f64880a2c20 0x7f64880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:39.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.552+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f649400f020 con 0x7f649c107590 2024-04-22T01:10:39.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.552+0000 7f649ad9d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f64880a2c20 0x7f64880a50e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6484005950 tx=0x7f648400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:39.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.556+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f649402b030 con 0x7f649c107590 2024-04-22T01:10:39.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.810+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f649c02db30 con 0x7f649c107590 2024-04-22T01:10:39.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:39 smithi038 ceph-mon[28043]: pgmap v300: 129 pgs: 5 creating+peering, 124 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-04-22T01:10:39.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.811+0000 7f6493fff700 1 -- 172.21.15.38:0/3568912750 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f649409d350 con 0x7f649c107590 2024-04-22T01:10:39.812 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:10:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f64880a2c20 msgr2=0x7f64880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f64880a2c20 0x7f64880a50e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6484005950 tx=0x7f648400a300 comp rx=0 tx=0).stop 2024-04-22T01:10:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 msgr2=0x7f649c074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c074fb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f64940062a0 tx=0x7f649400be60 comp rx=0 tx=0).stop 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 shutdown_connections 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f64880a2c20 0x7f64880a50e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f649c107590 0x7f649c074fb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f649c102510 0x7f649c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 --2- 172.21.15.38:0/3568912750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f649c102ec0 0x7f649c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 >> 172.21.15.38:0/3568912750 conn(0x7f649c0793a0 msgr2=0x7f649c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 shutdown_connections 2024-04-22T01:10:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:39.813+0000 7f64a124d700 1 -- 172.21.15.38:0/3568912750 wait complete. 2024-04-22T01:10:40.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:39 smithi203 ceph-mon[31825]: pgmap v300: 129 pgs: 5 creating+peering, 124 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-04-22T01:10:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:39 smithi111 ceph-mon[31057]: pgmap v300: 129 pgs: 5 creating+peering, 124 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-04-22T01:10:40.605 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-22T01:10:40.605 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:10:40.605 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2024-04-22T01:10:40.606 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-22T01:10:40.606 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:40.606 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-22T01:10:40.606 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:10:40.606 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:10:40.606 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-22T01:10:40.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:40.606 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2024-04-22T01:10:40.625 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:10:40.625 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f0 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f1 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:10:40.626 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f0 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 42085sec preferred_lft 42085sec 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:10:40.627 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f1 2024-04-22T01:10:40.628 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T01:10:40.628 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:10:40.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:10:40.820 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:40 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3568912750' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T01:10:40.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:40.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:40.829 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:10:40.855 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-04-22T01:10:40.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:40.857 DEBUG:teuthology.orchestra.run.smithi038:> route 2024-04-22T01:10:40.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:40 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3568912750' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T01:10:40.930 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2024-04-22T01:10:40.930 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:10:40.931 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-22T01:10:40.931 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-22T01:10:40.931 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:10:40.931 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:10:40.931 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:10:40.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-22T01:10:40.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-22T01:10:40.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-22T01:10:40.932 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:10:41.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:40 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3568912750' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T01:10:41.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:10:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:41.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:41.220 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T01:10:41.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:41.236 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2024-04-22T01:10:41.291 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:10:41.292 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:10:41.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:41.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-22T01:10:41.292 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:10:41.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:10:41.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:41.412 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-22T01:10:41.412 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-22T01:10:41.413 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:10:41.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:10:41.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:41.669 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:10:41.669 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:10:41.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2024-04-22T01:10:41.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T01:10:41.670 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:10:41.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:10:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:41.752 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-22T01:10:41.752 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:10:41.752 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:41.768 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-22T01:10:41.768 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:10:41.768 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:41.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2024-04-22T01:10:41.889 DEBUG:teuthology.orchestra.run.smithi038:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:10:41.903 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:41 smithi038 ceph-mon[28043]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 4 op/s 2024-04-22T01:10:41.918 INFO:teuthology.orchestra.run.smithi038.stdout:/proc 2024-04-22T01:10:41.918 INFO:teuthology.orchestra.run.smithi038.stdout:/sys 2024-04-22T01:10:41.918 INFO:teuthology.orchestra.run.smithi038.stdout:/dev 2024-04-22T01:10:41.918 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/security 2024-04-22T01:10:41.918 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/shm 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/pts 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/run 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/cgroup 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/pstore 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/bpf 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/ 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/selinux 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/hugepages 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/mqueue 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/debug 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/tracing 2024-04-22T01:10:41.919 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/config 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/fuse/connections 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/1000 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/ceph 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/d997a2071907211a9c5fe04741ea19a872bd38c45c8441e0446c6bade2b5b392/merged 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/4220c432a7c1fd31d2116e6ebbc8312eafb8d84362eec48436a98ad3d4814cb2/merged 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/0 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:10:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/70a4529e298f2e557c473605dde5fe4170628961f6ea4a797548042e893e0a0a/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b9f3b6acd27cbb40c4cbdae1fcf325b5a2345b9ebdfb785ff27fb5dafdfe890f/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/bb3205a57e70a35d236615cfb8d013c135c45d819e10e83780442a65694f032a/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b40714f8fbda9e1027dfd188c704c915ab2ca9e7e83a353ed160219a0a30b5b5/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ff0c7596930d6da07fc6573c3824132ec583abeb503842cd591fbcec53d21b3d/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/a3d0552e66a066692067dd384c3745cbb1e81c1ba8285ad1431f2f4353ae0e94/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ddc56f52a7bee6c282d431d5670bd3c566ec4da3e65a19a7a7f07f9773b05569/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ce7bef6c018cb300ce727129f12673cbfddf53cc5f12a7ca7ece2a9f0863db75/merged 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:41.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:41.922 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2024-04-22T01:10:41.974 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T01:10:41.974 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-22T01:10:42.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:41 smithi203 ceph-mon[31825]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 4 op/s 2024-04-22T01:10:42.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2024-04-22T01:10:42.040 DEBUG:teuthology.orchestra.run.smithi038:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/proc 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/sys 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/dev 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/security 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/shm 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/pts 2024-04-22T01:10:42.097 INFO:teuthology.orchestra.run.smithi038.stdout:/run 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/cgroup 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/pstore 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/bpf 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/ 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/selinux 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/hugepages 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/mqueue 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/debug 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/tracing 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/config 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/fuse/connections 2024-04-22T01:10:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/1000 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/ceph 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/d997a2071907211a9c5fe04741ea19a872bd38c45c8441e0446c6bade2b5b392/merged 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/4220c432a7c1fd31d2116e6ebbc8312eafb8d84362eec48436a98ad3d4814cb2/merged 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/0 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/70a4529e298f2e557c473605dde5fe4170628961f6ea4a797548042e893e0a0a/merged 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b9f3b6acd27cbb40c4cbdae1fcf325b5a2345b9ebdfb785ff27fb5dafdfe890f/merged 2024-04-22T01:10:42.099 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/bb3205a57e70a35d236615cfb8d013c135c45d819e10e83780442a65694f032a/merged 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b40714f8fbda9e1027dfd188c704c915ab2ca9e7e83a353ed160219a0a30b5b5/merged 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ff0c7596930d6da07fc6573c3824132ec583abeb503842cd591fbcec53d21b3d/merged 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/a3d0552e66a066692067dd384c3745cbb1e81c1ba8285ad1431f2f4353ae0e94/merged 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ddc56f52a7bee6c282d431d5670bd3c566ec4da3e65a19a7a7f07f9773b05569/merged 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ce7bef6c018cb300ce727129f12673cbfddf53cc5f12a7ca7ece2a9f0863db75/merged 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:42.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:42.100 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2024-04-22T01:10:42.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:41 smithi111 ceph-mon[31057]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 4 op/s 2024-04-22T01:10:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:2024-04-22T01:10:42.186+0000 7fce70441480 -1 init, newargv = 0x55a8f376f170 newargc=15 2024-04-22T01:10:42.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[71365]: starting ceph client 2024-04-22T01:10:42.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[71365]: starting fuse 2024-04-22T01:10:43.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2024-04-22T01:10:43.285 DEBUG:teuthology.orchestra.run.smithi038:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T01:10:43.344 INFO:teuthology.orchestra.run.smithi038.stdout:/proc 2024-04-22T01:10:43.344 INFO:teuthology.orchestra.run.smithi038.stdout:/sys 2024-04-22T01:10:43.344 INFO:teuthology.orchestra.run.smithi038.stdout:/dev 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/security 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/shm 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/pts 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/run 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/cgroup 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/pstore 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/bpf 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/ 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/selinux 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/hugepages 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/mqueue 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/debug 2024-04-22T01:10:43.345 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/tracing 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/config 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/fuse/connections 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/1000 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/ceph 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/d997a2071907211a9c5fe04741ea19a872bd38c45c8441e0446c6bade2b5b392/merged 2024-04-22T01:10:43.346 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/4220c432a7c1fd31d2116e6ebbc8312eafb8d84362eec48436a98ad3d4814cb2/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/0 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/70a4529e298f2e557c473605dde5fe4170628961f6ea4a797548042e893e0a0a/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b9f3b6acd27cbb40c4cbdae1fcf325b5a2345b9ebdfb785ff27fb5dafdfe890f/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/bb3205a57e70a35d236615cfb8d013c135c45d819e10e83780442a65694f032a/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b40714f8fbda9e1027dfd188c704c915ab2ca9e7e83a353ed160219a0a30b5b5/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ff0c7596930d6da07fc6573c3824132ec583abeb503842cd591fbcec53d21b3d/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/a3d0552e66a066692067dd384c3745cbb1e81c1ba8285ad1431f2f4353ae0e94/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ddc56f52a7bee6c282d431d5670bd3c566ec4da3e65a19a7a7f07f9773b05569/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/ce7bef6c018cb300ce727129f12673cbfddf53cc5f12a7ca7ece2a9f0863db75/merged 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:10:43.347 INFO:teuthology.orchestra.run.smithi038.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:43.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:43.348 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2024-04-22T01:10:43.402 INFO:teuthology.orchestra.run.smithi038.stdout:115 2024-04-22T01:10:43.403 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo stdin-killer -- python3 -c ' 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> import glob 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> import re 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> import os 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> import subprocess 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> def _find_admin_socket(client_name): 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> files = glob.glob(asok_path) 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-22T01:10:43.403 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> # Given a non-glob path, it better be there 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> if "*" not in asok_path: 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> assert(len(files) == 1) 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> return files[0] 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> for f in files: 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> contents = proc_f.read() 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> if mountpoint in contents: 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> return f 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-22T01:10:43.404 DEBUG:teuthology.orchestra.run.smithi038:> print(_find_admin_socket("client.admin")) 2024-04-22T01:10:43.405 DEBUG:teuthology.orchestra.run.smithi038:> ' 2024-04-22T01:10:43.532 INFO:teuthology.orchestra.run.smithi038.stdout:/var/run/ceph/ceph-client.admin.71365.asok 2024-04-22T01:10:43.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:43.543 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71365.asok 2024-04-22T01:10:43.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:43.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71365.asok status 2024-04-22T01:10:43.682 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:10:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "metadata": { 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "entity_id": "admin", 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": "smithi038", 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "pid": "71365", 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "root": "/" 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dentry_count": 0, 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dentry_pinned_count": 0, 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 14745, 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "inst": { 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "name": { 2024-04-22T01:10:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "client", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 14745 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": { 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 354074249 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": { 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "192.168.0.1:0", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 354074249 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "inst_str": "client.14745 192.168.0.1:0/354074249", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "addr_str": "192.168.0.1:0/354074249", 2024-04-22T01:10:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "inode_count": 1, 2024-04-22T01:10:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_epoch": 17, 2024-04-22T01:10:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_epoch": 89, 2024-04-22T01:10:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_epoch_barrier": 0, 2024-04-22T01:10:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklisted": false, 2024-04-22T01:10:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs" 2024-04-22T01:10:43.685 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:10:43.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:43.691 DEBUG:teuthology.orchestra.run.smithi038:> stat --file-system '--printf=%T 2024-04-22T01:10:43.691 DEBUG:teuthology.orchestra.run.smithi038:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:43.707 INFO:teuthology.orchestra.run.smithi038.stdout:fuseblk 2024-04-22T01:10:43.707 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:43.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:43.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:43.773 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-22T01:10:43.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T01:10:43.882 INFO:teuthology.orchestra.run.smithi038.stdout:# file: . 2024-04-22T01:10:43.882 INFO:teuthology.orchestra.run.smithi038.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-22T01:10:43.882 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:43.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:10:43.889 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:10:43.889 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:10:43.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:43.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:43 smithi038 ceph-mon[28043]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-22T01:10:44.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:43 smithi203 ceph-mon[31825]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-22T01:10:44.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 -- 172.21.15.38:0/2441375801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 msgr2=0x7f3be4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 --2- 172.21.15.38:0/2441375801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4101d10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009900 tx=0x7f3bd002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:44.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 -- 172.21.15.38:0/2441375801 shutdown_connections 2024-04-22T01:10:44.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 --2- 172.21.15.38:0/2441375801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be4102250 0x7f3be4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 --2- 172.21.15.38:0/2441375801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4101d10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 --2- 172.21.15.38:0/2441375801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be41006a0 0x7f3be4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 -- 172.21.15.38:0/2441375801 >> 172.21.15.38:0/2441375801 conn(0x7f3be40fbc30 msgr2=0x7f3be40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 -- 172.21.15.38:0/2441375801 shutdown_connections 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.047+0000 7f3beafaf640 1 -- 172.21.15.38:0/2441375801 wait complete. 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 Processor -- start 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 -- start start 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be41006a0 0x7f3be4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4102250 0x7f3be419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3be410c300 con 0x7f3be4101890 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3be410c180 con 0x7f3be41006a0 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3beafaf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3be410c480 con 0x7f3be4102250 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49370/0 (socket says 172.21.15.38:49370) 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 -- 172.21.15.38:0/3487679162 learned_addr learned my addr 172.21.15.38:0/3487679162 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be954c640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4102250 0x7f3be419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be8d4b640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be41006a0 0x7f3be4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 -- 172.21.15.38:0/3487679162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4102250 msgr2=0x7f3be419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4102250 0x7f3be419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 -- 172.21.15.38:0/3487679162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be41006a0 msgr2=0x7f3be4197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be41006a0 0x7f3be4197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 -- 172.21.15.38:0/3487679162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be419f660 con 0x7f3be4101890 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be8d4b640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be41006a0 0x7f3be4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.048+0000 7f3be3fff640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4197f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009230 tx=0x7f3bd0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.049+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0041070 con 0x7f3be4101890 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.049+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be419f8f0 con 0x7f3be4101890 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.049+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd00054e0 con 0x7f3be4101890 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.049+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0034440 con 0x7f3be4101890 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.049+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be419fde0 con 0x7f3be4101890 2024-04-22T01:10:44.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.050+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be419ffe0 con 0x7f3be4101890 2024-04-22T01:10:44.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.051+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3bd0044070 con 0x7f3be4101890 2024-04-22T01:10:44.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.051+0000 7f3be1ffb640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3bd40a2b90 0x7f3bd40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.052+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3bd00f1460 con 0x7f3be4101890 2024-04-22T01:10:44.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.052+0000 7f3be8d4b640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3bd40a2b90 0x7f3bd40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.052+0000 7f3be8d4b640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3bd40a2b90 0x7f3bd40a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3be41016f0 tx=0x7f3bd8005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.055+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3bd00be180 con 0x7f3be4101890 2024-04-22T01:10:44.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:43 smithi111 ceph-mon[31057]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-22T01:10:44.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.241+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3be41a0370 con 0x7f3be4101890 2024-04-22T01:10:44.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.243+0000 7f3be1ffb640 1 -- 172.21.15.38:0/3487679162 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f3bd00bda80 con 0x7f3be4101890 2024-04-22T01:10:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:44.244 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3bd40a2b90 msgr2=0x7f3bd40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3bd40a2b90 0x7f3bd40a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3be41016f0 tx=0x7f3bd8005db0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 msgr2=0x7f3be4197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4197f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009230 tx=0x7f3bd0005dd0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 shutdown_connections 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3bd40a2b90 0x7f3bd40a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4102250 0x7f3be419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3be4101890 0x7f3be4197f80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 --2- 172.21.15.38:0/3487679162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3be41006a0 0x7f3be4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 >> 172.21.15.38:0/3487679162 conn(0x7f3be40fbc30 msgr2=0x7f3be40fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 shutdown_connections 2024-04-22T01:10:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.245+0000 7f3beafaf640 1 -- 172.21.15.38:0/3487679162 wait complete. 2024-04-22T01:10:44.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:10:44.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.361+0000 7f3b7c46f640 1 -- 172.21.15.38:0/2947874732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74102250 msgr2=0x7f3b74108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.361+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/2947874732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74102250 0x7f3b74108ae0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c009ca0 tx=0x7f3b6c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- 172.21.15.38:0/2947874732 shutdown_connections 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/2947874732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74102250 0x7f3b74108ae0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/2947874732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b74101890 0x7f3b74101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/2947874732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b741006a0 0x7f3b74100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- 172.21.15.38:0/2947874732 >> 172.21.15.38:0/2947874732 conn(0x7f3b740fbc30 msgr2=0x7f3b740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- 172.21.15.38:0/2947874732 shutdown_connections 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- 172.21.15.38:0/2947874732 wait complete. 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 Processor -- start 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- start start 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b741006a0 0x7f3b74197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 0x7f3b74197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b74102250 0x7f3b7419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7410d440 con 0x7f3b74101890 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7410d2c0 con 0x7f3b741006a0 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b7c46f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7410d5c0 con 0x7f3b74102250 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.362+0000 7f3b79a0a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 0x7f3b74197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 0x7f3b74197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49390/0 (socket says 172.21.15.38:49390) 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 -- 172.21.15.38:0/739462606 learned_addr learned my addr 172.21.15.38:0/739462606 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b7aa0c640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b74102250 0x7f3b7419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b7a20b640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b741006a0 0x7f3b74197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 -- 172.21.15.38:0/739462606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b74102250 msgr2=0x7f3b7419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b74102250 0x7f3b7419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 -- 172.21.15.38:0/739462606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b741006a0 msgr2=0x7f3b74197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b741006a0 0x7f3b74197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 -- 172.21.15.38:0/739462606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7419f620 con 0x7f3b74101890 2024-04-22T01:10:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b79a0a640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 0x7f3b74197f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400d770 tx=0x7f3b6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64017070 con 0x7f3b74101890 2024-04-22T01:10:44.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b64005500 con 0x7f3b74101890 2024-04-22T01:10:44.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7419f910 con 0x7f3b74101890 2024-04-22T01:10:44.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64004e00 con 0x7f3b74101890 2024-04-22T01:10:44.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.363+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7419fe10 con 0x7f3b74101890 2024-04-22T01:10:44.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.364+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b741079c0 con 0x7f3b74101890 2024-04-22T01:10:44.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.366+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3b64004fa0 con 0x7f3b74101890 2024-04-22T01:10:44.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.366+0000 7f3b6b7fe640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3b600a2b90 0x7f3b600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.366+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3b640cefd0 con 0x7f3b74101890 2024-04-22T01:10:44.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.366+0000 7f3b7a20b640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3b600a2b90 0x7f3b600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.367+0000 7f3b7a20b640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3b600a2b90 0x7f3b600a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005fd0 tx=0x7f3b7000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.370+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3b6409b6a0 con 0x7f3b74101890 2024-04-22T01:10:44.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.555+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b7402d210 con 0x7f3b74101890 2024-04-22T01:10:44.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.555+0000 7f3b6b7fe640 1 -- 172.21.15.38:0/739462606 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f3b6409f9a0 con 0x7f3b74101890 2024-04-22T01:10:44.556 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3b600a2b90 msgr2=0x7f3b600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3b600a2b90 0x7f3b600a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005fd0 tx=0x7f3b7000b410 comp rx=0 tx=0).stop 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 msgr2=0x7f3b74197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 0x7f3b74197f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400d770 tx=0x7f3b6400dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 shutdown_connections 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3b600a2b90 0x7f3b600a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b74102250 0x7f3b7419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b74101890 0x7f3b74197f40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 --2- 172.21.15.38:0/739462606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b741006a0 0x7f3b74197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.557+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 >> 172.21.15.38:0/739462606 conn(0x7f3b740fbc30 msgr2=0x7f3b74104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.558+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 shutdown_connections 2024-04-22T01:10:44.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.558+0000 7f3b7c46f640 1 -- 172.21.15.38:0/739462606 wait complete. 2024-04-22T01:10:44.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:10:44.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3229893418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c8102990 msgr2=0x7f69c8102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3229893418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c8102990 0x7f69c8102e10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009900 tx=0x7f69b402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3229893418 shutdown_connections 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3229893418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8103350 0x7f69c8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3229893418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c8102990 0x7f69c8102e10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3229893418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69c81017a0 0x7f69c8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3229893418 >> 172.21.15.38:0/3229893418 conn(0x7f69c80fcd10 msgr2=0x7f69c80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3229893418 shutdown_connections 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.673+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3229893418 wait complete. 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 Processor -- start 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 -- start start 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 0x7f69c81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8102990 0x7f69c8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69c8103350 0x7f69c819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810d400 con 0x7f69c81017a0 2024-04-22T01:10:44.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810d280 con 0x7f69c8103350 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69ceaf4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810d580 con 0x7f69c8102990 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 0x7f69c81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 0x7f69c81979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49400/0 (socket says 172.21.15.38:49400) 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 -- 172.21.15.38:0/3642654683 learned_addr learned my addr 172.21.15.38:0/3642654683 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69c7fff640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8102990 0x7f69c8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cd091640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69c8103350 0x7f69c819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 -- 172.21.15.38:0/3642654683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8102990 msgr2=0x7f69c8197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8102990 0x7f69c8197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 -- 172.21.15.38:0/3642654683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69c8103350 msgr2=0x7f69c819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69c8103350 0x7f69c819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.674+0000 7f69cc890640 1 -- 172.21.15.38:0/3642654683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c819f600 con 0x7f69c81017a0 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69c7fff640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8102990 0x7f69c8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69cc890640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 0x7f69c81979e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c970 tx=0x7f69bc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc007c10 con 0x7f69c81017a0 2024-04-22T01:10:44.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c819f8f0 con 0x7f69c81017a0 2024-04-22T01:10:44.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69bc007db0 con 0x7f69c81017a0 2024-04-22T01:10:44.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc00f8a0 con 0x7f69c81017a0 2024-04-22T01:10:44.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.675+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c819fe10 con 0x7f69c81017a0 2024-04-22T01:10:44.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.676+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c81089f0 con 0x7f69c81017a0 2024-04-22T01:10:44.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.677+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f69bc00fa60 con 0x7f69c81017a0 2024-04-22T01:10:44.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.678+0000 7f69c5ffb640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f69b80a2b90 0x7f69b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.678+0000 7f69c7fff640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f69b80a2b90 0x7f69b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.678+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f69bc0cf630 con 0x7f69c81017a0 2024-04-22T01:10:44.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.679+0000 7f69c7fff640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f69b80a2b90 0x7f69b80a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006010 tx=0x7f69b4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.686+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f69bc09c380 con 0x7f69c81017a0 2024-04-22T01:10:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.872+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69c8062340 con 0x7f69c81017a0 2024-04-22T01:10:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.872+0000 7f69c5ffb640 1 -- 172.21.15.38:0/3642654683 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f69bc09bc80 con 0x7f69c81017a0 2024-04-22T01:10:44.872 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f69b80a2b90 msgr2=0x7f69b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f69b80a2b90 0x7f69b80a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006010 tx=0x7f69b4000bc0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 msgr2=0x7f69c81979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 0x7f69c81979e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c970 tx=0x7f69bc00ce30 comp rx=0 tx=0).stop 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 shutdown_connections 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f69b80a2b90 0x7f69b80a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69c8103350 0x7f69c819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c8102990 0x7f69c8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 --2- 172.21.15.38:0/3642654683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69c81017a0 0x7f69c81979e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 >> 172.21.15.38:0/3642654683 conn(0x7f69c80fcd10 msgr2=0x7f69c8105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.874+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 shutdown_connections 2024-04-22T01:10:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.875+0000 7f69ceaf4640 1 -- 172.21.15.38:0/3642654683 wait complete. 2024-04-22T01:10:44.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:44 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3487679162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:44.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:44 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/739462606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:44.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:10:44.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 -- 172.21.15.38:0/2484895835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0102980 msgr2=0x7f48a0104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/2484895835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0102980 0x7f48a0104df0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4890009900 tx=0x7f489002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 -- 172.21.15.38:0/2484895835 shutdown_connections 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/2484895835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48a0105330 0x7f48a01057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/2484895835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0102980 0x7f48a0104df0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/2484895835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a0100010 0x7f48a0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 -- 172.21.15.38:0/2484895835 >> 172.21.15.38:0/2484895835 conn(0x7f48a00fbc30 msgr2=0x7f48a00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 -- 172.21.15.38:0/2484895835 shutdown_connections 2024-04-22T01:10:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.990+0000 7f48a7cc6640 1 -- 172.21.15.38:0/2484895835 wait complete. 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 Processor -- start 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 -- start start 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 0x7f48a019e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a0102980 0x7f48a019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48a0105330 0x7f48a01a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48a0113a90 con 0x7f48a0100010 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48a0113910 con 0x7f48a0102980 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a7cc6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48a0113c10 con 0x7f48a0105330 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 0x7f48a019e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 0x7f48a019e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49422/0 (socket says 172.21.15.38:49422) 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 -- 172.21.15.38:0/1486469484 learned_addr learned my addr 172.21.15.38:0/1486469484 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a6263640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48a0105330 0x7f48a01a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 -- 172.21.15.38:0/1486469484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48a0105330 msgr2=0x7f48a01a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48a0105330 0x7f48a01a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 -- 172.21.15.38:0/1486469484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a0102980 msgr2=0x7f48a019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.991+0000 7f48a5a62640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a0102980 0x7f48a019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f48a5a62640 1 -- 172.21.15.38:0/1486469484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a01a5d30 con 0x7f48a0100010 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f48a5a62640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 0x7f48a019e110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b4a0 tx=0x7f489c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c014070 con 0x7f48a0100010 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f489c010af0 con 0x7f48a0100010 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a01a5fc0 con 0x7f48a0100010 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c017d10 con 0x7f48a0100010 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.992+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a01a6530 con 0x7f48a0100010 2024-04-22T01:10:44.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.993+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a00fd720 con 0x7f48a0100010 2024-04-22T01:10:44.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.994+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f489c017070 con 0x7f48a0100010 2024-04-22T01:10:44.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.995+0000 7f4896ffd640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f488c0a2c10 0x7f488c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:44.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.995+0000 7f48a5261640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f488c0a2c10 0x7f488c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:44.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.995+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f489c0ce660 con 0x7f48a0100010 2024-04-22T01:10:44.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.995+0000 7f48a5261640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f488c0a2c10 0x7f488c0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f489002c710 tx=0x7f4890000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:44.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:44.999+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f489c09b320 con 0x7f48a0100010 2024-04-22T01:10:45.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:44 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3487679162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:45.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:44 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/739462606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:45.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:44 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3487679162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:45.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:44 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/739462606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.183+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f48a00623e0 con 0x7f48a0100010 2024-04-22T01:10:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.183+0000 7f4896ffd640 1 -- 172.21.15.38:0/1486469484 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f489c09ac20 con 0x7f48a0100010 2024-04-22T01:10:45.183 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:10:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f488c0a2c10 msgr2=0x7f488c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f488c0a2c10 0x7f488c0a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f489002c710 tx=0x7f4890000bc0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 msgr2=0x7f48a019e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 0x7f48a019e110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b4a0 tx=0x7f489c00b960 comp rx=0 tx=0).stop 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 shutdown_connections 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f488c0a2c10 0x7f488c0a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48a0105330 0x7f48a01a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a0102980 0x7f48a019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 --2- 172.21.15.38:0/1486469484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a0100010 0x7f48a019e110 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 >> 172.21.15.38:0/1486469484 conn(0x7f48a00fbc30 msgr2=0x7f48a0100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 shutdown_connections 2024-04-22T01:10:45.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.185+0000 7f48a7cc6640 1 -- 172.21.15.38:0/1486469484 wait complete. 2024-04-22T01:10:45.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:10:45.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 -- 172.21.15.38:0/2472766557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 msgr2=0x7f5188101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 --2- 172.21.15.38:0/2472766557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188101d10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5170009900 tx=0x7f517002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:45.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 -- 172.21.15.38:0/2472766557 shutdown_connections 2024-04-22T01:10:45.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 --2- 172.21.15.38:0/2472766557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5188102250 0x7f5188108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 --2- 172.21.15.38:0/2472766557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188101d10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 --2- 172.21.15.38:0/2472766557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51881006a0 0x7f5188100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 -- 172.21.15.38:0/2472766557 >> 172.21.15.38:0/2472766557 conn(0x7f51880fbc30 msgr2=0x7f51880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 -- 172.21.15.38:0/2472766557 shutdown_connections 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.300+0000 7f518eee6640 1 -- 172.21.15.38:0/2472766557 wait complete. 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 Processor -- start 2024-04-22T01:10:45.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 -- start start 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51881006a0 0x7f518810b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5188102250 0x7f5188109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f518810d4a0 con 0x7f5188101890 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f518810d320 con 0x7f5188102250 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518eee6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f518810d620 con 0x7f51881006a0 2024-04-22T01:10:45.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49444/0 (socket says 172.21.15.38:49444) 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 -- 172.21.15.38:0/1510603246 learned_addr learned my addr 172.21.15.38:0/1510603246 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518cc82640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51881006a0 0x7f518810b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f518d483640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5188102250 0x7f5188109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 -- 172.21.15.38:0/1510603246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51881006a0 msgr2=0x7f518810b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51881006a0 0x7f518810b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 -- 172.21.15.38:0/1510603246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5188102250 msgr2=0x7f5188109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5188102250 0x7f5188109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 -- 172.21.15.38:0/1510603246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518810a430 con 0x7f5188101890 2024-04-22T01:10:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5187fff640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188109660 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5170000c00 tx=0x7f5170005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.301+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170043020 con 0x7f5188101890 2024-04-22T01:10:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.302+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51700054e0 con 0x7f5188101890 2024-04-22T01:10:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.302+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51881a1bb0 con 0x7f5188101890 2024-04-22T01:10:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.302+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170003e60 con 0x7f5188101890 2024-04-22T01:10:45.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.302+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51881a2150 con 0x7f5188101890 2024-04-22T01:10:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.303+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5188107a10 con 0x7f5188101890 2024-04-22T01:10:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.304+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5170046020 con 0x7f5188101890 2024-04-22T01:10:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.304+0000 7f5185ffb640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f51740a2c10 0x7f51740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.305+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51700f1520 con 0x7f5188101890 2024-04-22T01:10:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.305+0000 7f518cc82640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f51740a2c10 0x7f51740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.305+0000 7f518cc82640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f51740a2c10 0x7f51740a50d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5178005fd0 tx=0x7f5178005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:45.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.309+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f517004d020 con 0x7f5188101890 2024-04-22T01:10:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.492+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5188062340 con 0x7f5188101890 2024-04-22T01:10:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.493+0000 7f5185ffb640 1 -- 172.21.15.38:0/1510603246 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f51700be240 con 0x7f5188101890 2024-04-22T01:10:45.493 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f51740a2c10 msgr2=0x7f51740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f51740a2c10 0x7f51740a50d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5178005fd0 tx=0x7f5178005850 comp rx=0 tx=0).stop 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 msgr2=0x7f5188109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188109660 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5170000c00 tx=0x7f5170005dd0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 shutdown_connections 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f51740a2c10 0x7f51740a50d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5188102250 0x7f5188109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5188101890 0x7f5188109660 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 --2- 172.21.15.38:0/1510603246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51881006a0 0x7f518810b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 >> 172.21.15.38:0/1510603246 conn(0x7f51880fbc30 msgr2=0x7f5188104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:45.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 shutdown_connections 2024-04-22T01:10:45.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.495+0000 7f518eee6640 1 -- 172.21.15.38:0/1510603246 wait complete. 2024-04-22T01:10:45.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 -- 172.21.15.38:0/3482642008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 msgr2=0x7f2730100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 --2- 172.21.15.38:0/3482642008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730100ac0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f272c009900 tx=0x7f272c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 -- 172.21.15.38:0/3482642008 shutdown_connections 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 --2- 172.21.15.38:0/3482642008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730102250 0x7f2730108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 --2- 172.21.15.38:0/3482642008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2730101890 0x7f2730101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 --2- 172.21.15.38:0/3482642008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730100ac0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 -- 172.21.15.38:0/3482642008 >> 172.21.15.38:0/3482642008 conn(0x7f27300fbc30 msgr2=0x7f27300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 -- 172.21.15.38:0/3482642008 shutdown_connections 2024-04-22T01:10:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.610+0000 7f2738088640 1 -- 172.21.15.38:0/3482642008 wait complete. 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 Processor -- start 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 -- start start 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730104770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730101890 0x7f2730102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2730102250 0x7f2730103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f273010c320 con 0x7f27301006a0 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f273010c1a0 con 0x7f2730101890 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2738088640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f273010c4a0 con 0x7f2730102250 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730104770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730104770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49450/0 (socket says 172.21.15.38:49450) 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 -- 172.21.15.38:0/4078990631 learned_addr learned my addr 172.21.15.38:0/4078990631 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735623640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730101890 0x7f2730102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 -- 172.21.15.38:0/4078990631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2730102250 msgr2=0x7f2730103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2730102250 0x7f2730103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 -- 172.21.15.38:0/4078990631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730101890 msgr2=0x7f2730102e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730101890 0x7f2730102e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 -- 172.21.15.38:0/4078990631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2730103c00 con 0x7f27301006a0 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735623640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730101890 0x7f2730102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2735e24640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730104770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f272c005ab0 tx=0x7f272c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.611+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c041070 con 0x7f27301006a0 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.612+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f272c0054e0 con 0x7f27301006a0 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.612+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27301a1c50 con 0x7f27301006a0 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.612+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c034440 con 0x7f27301006a0 2024-04-22T01:10:45.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.612+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27301a2110 con 0x7f27301006a0 2024-04-22T01:10:45.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.612+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27301a22e0 con 0x7f27301006a0 2024-04-22T01:10:45.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.614+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f272c044070 con 0x7f27301006a0 2024-04-22T01:10:45.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.614+0000 7f2726ffd640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f271c0a2b90 0x7f271c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.615+0000 7f2735623640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f271c0a2b90 0x7f271c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.615+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f272c0f0f40 con 0x7f27301006a0 2024-04-22T01:10:45.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.615+0000 7f2735623640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f271c0a2b90 0x7f271c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f27301044e0 tx=0x7f2720005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.619+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f272c0bdc60 con 0x7f27301006a0 2024-04-22T01:10:45.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.820+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f27301a2670 con 0x7f27301006a0 2024-04-22T01:10:45.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.820+0000 7f2726ffd640 1 -- 172.21.15.38:0/4078990631 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+589 (secure 0 0 0) 0x7f272c0bd560 con 0x7f27301006a0 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3063830265 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3063830265 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2675569647 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2608977615 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3674504753 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3674504753 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/799365618 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3519840670 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/845318699 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/687303682 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:45.821 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2024-04-22T01:10:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f271c0a2b90 msgr2=0x7f271c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f271c0a2b90 0x7f271c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f27301044e0 tx=0x7f2720005e50 comp rx=0 tx=0).stop 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 msgr2=0x7f2730104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730104770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f272c005ab0 tx=0x7f272c005dc0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 shutdown_connections 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f271c0a2b90 0x7f271c0a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2730102250 0x7f2730103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2730101890 0x7f2730102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 --2- 172.21.15.38:0/4078990631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27301006a0 0x7f2730104770 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 >> 172.21.15.38:0/4078990631 conn(0x7f27300fbc30 msgr2=0x7f2730069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 shutdown_connections 2024-04-22T01:10:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.822+0000 7f2738088640 1 -- 172.21.15.38:0/4078990631 wait complete. 2024-04-22T01:10:45.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:45 smithi038 ceph-mon[28043]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 223 B/s wr, 0 op/s 2024-04-22T01:10:45.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:45 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3642654683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:10:45.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:45 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1486469484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:10:45.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:45 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1510603246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:10:45.835 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:10:45.835 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:10:45.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 -- 172.21.15.38:0/1937762926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 msgr2=0x7f9078101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 --2- 172.21.15.38:0/1937762926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078101bd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9068009900 tx=0x7f906802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 -- 172.21.15.38:0/1937762926 shutdown_connections 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 --2- 172.21.15.38:0/1937762926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9078103360 0x7f9078109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 --2- 172.21.15.38:0/1937762926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90781029a0 0x7f9078102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 --2- 172.21.15.38:0/1937762926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078101bd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 -- 172.21.15.38:0/1937762926 >> 172.21.15.38:0/1937762926 conn(0x7f90780fcd40 msgr2=0x7f90780ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 -- 172.21.15.38:0/1937762926 shutdown_connections 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.992+0000 7f907d635640 1 -- 172.21.15.38:0/1937762926 wait complete. 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 Processor -- start 2024-04-22T01:10:45.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 -- start start 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781029a0 0x7f9078198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9078103360 0x7f907819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f907810d400 con 0x7f90781017b0 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f907810d280 con 0x7f9078103360 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f907d635640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f907810d580 con 0x7f90781029a0 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f9076ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f9076ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49472/0 (socket says 172.21.15.38:49472) 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f9076ffd640 1 -- 172.21.15.38:0/1607569923 learned_addr learned my addr 172.21.15.38:0/1607569923 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f90777fe640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9078103360 0x7f907819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.993+0000 7f906ffff640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781029a0 0x7f9078198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9076ffd640 1 -- 172.21.15.38:0/1607569923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781029a0 msgr2=0x7f9078198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9076ffd640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781029a0 0x7f9078198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9076ffd640 1 -- 172.21.15.38:0/1607569923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9078103360 msgr2=0x7f907819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9076ffd640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9078103360 0x7f907819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9076ffd640 1 -- 172.21.15.38:0/1607569923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907819f6f0 con 0x7f90781017b0 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f906ffff640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781029a0 0x7f9078198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9076ffd640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078197ad0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9068000c00 tx=0x7f9068005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068041070 con 0x7f90781017b0 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907819f980 con 0x7f90781017b0 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90680044d0 con 0x7f90781017b0 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068034440 con 0x7f90781017b0 2024-04-22T01:10:45.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.994+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907819fec0 con 0x7f90781017b0 2024-04-22T01:10:45.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.995+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9078108af0 con 0x7f90781017b0 2024-04-22T01:10:45.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.996+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9068044070 con 0x7f90781017b0 2024-04-22T01:10:45.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.997+0000 7f9074ff9640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90640a2b90 0x7f90640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:45.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.997+0000 7f906ffff640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90640a2b90 0x7f90640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:45.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.997+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f90680f0e20 con 0x7f90781017b0 2024-04-22T01:10:45.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:45.997+0000 7f906ffff640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90640a2b90 0x7f90640a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9078198fe0 tx=0x7f9060005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.001+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f90680bdb40 con 0x7f90781017b0 2024-04-22T01:10:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:45 smithi203 ceph-mon[31825]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 223 B/s wr, 0 op/s 2024-04-22T01:10:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:45 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3642654683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:45 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1486469484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:45 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1510603246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:45 smithi111 ceph-mon[31057]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 223 B/s wr, 0 op/s 2024-04-22T01:10:46.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:45 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3642654683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:45 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1486469484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:45 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1510603246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.188+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9078062340 con 0x7f90781017b0 2024-04-22T01:10:46.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.189+0000 7f9074ff9640 1 -- 172.21.15.38:0/1607569923 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f90680bd440 con 0x7f90781017b0 2024-04-22T01:10:46.190 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:46.190 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:46.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.191+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90640a2b90 msgr2=0x7f90640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.191+0000 7f907d635640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90640a2b90 0x7f90640a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9078198fe0 tx=0x7f9060005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:46.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.191+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 msgr2=0x7f9078197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.191+0000 7f907d635640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078197ad0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9068000c00 tx=0x7f9068005d50 comp rx=0 tx=0).stop 2024-04-22T01:10:46.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 shutdown_connections 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90640a2b90 0x7f90640a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9078103360 0x7f907819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781029a0 0x7f9078198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 --2- 172.21.15.38:0/1607569923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90781017b0 0x7f9078197ad0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 >> 172.21.15.38:0/1607569923 conn(0x7f90780fcd40 msgr2=0x7f9078105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 shutdown_connections 2024-04-22T01:10:46.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.192+0000 7f907d635640 1 -- 172.21.15.38:0/1607569923 wait complete. 2024-04-22T01:10:46.203 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:10:46.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:10:46.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 -- 172.21.15.38:0/3451506837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8102990 msgr2=0x7fd8c8102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 --2- 172.21.15.38:0/3451506837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8102990 0x7fd8c8102e10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009900 tx=0x7fd8b802bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 -- 172.21.15.38:0/3451506837 shutdown_connections 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 --2- 172.21.15.38:0/3451506837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 0x7fd8c8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 --2- 172.21.15.38:0/3451506837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8102990 0x7fd8c8102e10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 --2- 172.21.15.38:0/3451506837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c81017a0 0x7fd8c8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 -- 172.21.15.38:0/3451506837 >> 172.21.15.38:0/3451506837 conn(0x7fd8c80fcd10 msgr2=0x7fd8c80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 -- 172.21.15.38:0/3451506837 shutdown_connections 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.308+0000 7fd8cf775640 1 -- 172.21.15.38:0/3451506837 wait complete. 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 Processor -- start 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 -- start start 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 0x7fd8c8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c8102990 0x7fd8c8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 0x7fd8c819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c810d400 con 0x7fd8c81017a0 2024-04-22T01:10:46.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c810d280 con 0x7fd8c8103350 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cf775640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c810d580 con 0x7fd8c8102990 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 0x7fd8c8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 0x7fd8c8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49490/0 (socket says 172.21.15.38:49490) 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 -- 172.21.15.38:0/701443972 learned_addr learned my addr 172.21.15.38:0/701443972 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cdd12640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 0x7fd8c819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8ccd10640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c8102990 0x7fd8c8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 -- 172.21.15.38:0/701443972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c8102990 msgr2=0x7fd8c8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c8102990 0x7fd8c8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 -- 172.21.15.38:0/701443972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 msgr2=0x7fd8c819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 0x7fd8c819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cd511640 1 -- 172.21.15.38:0/701443972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c819f6f0 con 0x7fd8c81017a0 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8ccd10640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c8102990 0x7fd8c8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.309+0000 7fd8cdd12640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 0x7fd8c819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.310+0000 7fd8cd511640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 0x7fd8c8197ad0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400d950 tx=0x7fd8c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.310+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4017020 con 0x7fd8c81017a0 2024-04-22T01:10:46.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.310+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8c4009c30 con 0x7fd8c81017a0 2024-04-22T01:10:46.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.310+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c400f720 con 0x7fd8c81017a0 2024-04-22T01:10:46.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.310+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c819f9e0 con 0x7fd8c81017a0 2024-04-22T01:10:46.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.310+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c819ff00 con 0x7fd8c81017a0 2024-04-22T01:10:46.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.311+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c8108ae0 con 0x7fd8c81017a0 2024-04-22T01:10:46.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.312+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd8c401a020 con 0x7fd8c81017a0 2024-04-22T01:10:46.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.312+0000 7fd8be7fc640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd8b40a2b90 0x7fd8b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.313+0000 7fd8ccd10640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd8b40a2b90 0x7fd8b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.313+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd8c40cf1f0 con 0x7fd8c81017a0 2024-04-22T01:10:46.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.313+0000 7fd8ccd10640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd8b40a2b90 0x7fd8b40a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8006010 tx=0x7fd8b802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.317+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd8c4021020 con 0x7fd8c81017a0 2024-04-22T01:10:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.505+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd8c802d1f0 con 0x7fd8c81017a0 2024-04-22T01:10:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.506+0000 7fd8be7fc640 1 -- 172.21.15.38:0/701443972 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fd8c4009de0 con 0x7fd8c81017a0 2024-04-22T01:10:46.506 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:10:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd8b40a2b90 msgr2=0x7fd8b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd8b40a2b90 0x7fd8b40a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8006010 tx=0x7fd8b802c8a0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 msgr2=0x7fd8c8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 0x7fd8c8197ad0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400d950 tx=0x7fd8c400de10 comp rx=0 tx=0).stop 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 shutdown_connections 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd8b40a2b90 0x7fd8b40a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c8103350 0x7fd8c819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c8102990 0x7fd8c8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 --2- 172.21.15.38:0/701443972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c81017a0 0x7fd8c8197ad0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 >> 172.21.15.38:0/701443972 conn(0x7fd8c80fcd10 msgr2=0x7fd8c8105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 shutdown_connections 2024-04-22T01:10:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.508+0000 7fd8cf775640 1 -- 172.21.15.38:0/701443972 wait complete. 2024-04-22T01:10:46.519 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:10:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 -- 172.21.15.38:0/1445489331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704102230 msgr2=0x7f3704108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 --2- 172.21.15.38:0/1445489331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704102230 0x7f3704108ac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009ca0 tx=0x7f36fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 -- 172.21.15.38:0/1445489331 shutdown_connections 2024-04-22T01:10:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 --2- 172.21.15.38:0/1445489331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704102230 0x7f3704108ac0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 --2- 172.21.15.38:0/1445489331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 0x7f3704101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 --2- 172.21.15.38:0/1445489331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704100680 0x7f3704100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 -- 172.21.15.38:0/1445489331 >> 172.21.15.38:0/1445489331 conn(0x7f37040fbbf0 msgr2=0x7f37040fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 -- 172.21.15.38:0/1445489331 shutdown_connections 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.624+0000 7f370bb9a640 1 -- 172.21.15.38:0/1445489331 wait complete. 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 Processor -- start 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 -- start start 2024-04-22T01:10:46.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 0x7f3704069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 0x7f370419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704102230 0x7f370419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f370410c300 con 0x7f3704100680 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f370410c180 con 0x7f3704102230 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370bb9a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f370410c480 con 0x7f3704101870 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 0x7f3704069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 0x7f3704069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49510/0 (socket says 172.21.15.38:49510) 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 -- 172.21.15.38:0/361909813 learned_addr learned my addr 172.21.15.38:0/361909813 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370a137640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704102230 0x7f370419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 -- 172.21.15.38:0/361909813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 msgr2=0x7f370419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709135640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 0x7f370419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 0x7f370419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 -- 172.21.15.38:0/361909813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704102230 msgr2=0x7f370419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704102230 0x7f370419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 -- 172.21.15.38:0/361909813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370419f700 con 0x7f3704100680 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f370a137640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704102230 0x7f370419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709135640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 0x7f370419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:46.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.625+0000 7f3709936640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 0x7f3704069440 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f370000d770 tx=0x7f370000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.626+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700017070 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.626+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370419f990 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.626+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3700009c30 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.626+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700003dd0 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.626+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370419fee0 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.627+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3704068560 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.628+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3700003f70 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.628+0000 7f36faffd640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f36f00a2b90 0x7f36f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.628+0000 7f3709135640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f36f00a2b90 0x7f36f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.628+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f37000ced20 con 0x7f3704100680 2024-04-22T01:10:46.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.629+0000 7f3709135640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f36f00a2b90 0x7f36f00a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37040691b0 tx=0x7f36f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.633+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f370009ba40 con 0x7f3704100680 2024-04-22T01:10:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.818+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3704062370 con 0x7f3704100680 2024-04-22T01:10:46.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.818+0000 7f36faffd640 1 -- 172.21.15.38:0/361909813 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f3700009de0 con 0x7f3704100680 2024-04-22T01:10:46.819 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f36f00a2b90 msgr2=0x7f36f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f36f00a2b90 0x7f36f00a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37040691b0 tx=0x7f36f4005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 msgr2=0x7f3704069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 0x7f3704069440 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f370000d770 tx=0x7f370000dc30 comp rx=0 tx=0).stop 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 shutdown_connections 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f36f00a2b90 0x7f36f00a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3704102230 0x7f370419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3704101870 0x7f370419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 --2- 172.21.15.38:0/361909813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3704100680 0x7f3704069440 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.820+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 >> 172.21.15.38:0/361909813 conn(0x7f37040fbbf0 msgr2=0x7f3704104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.821+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 shutdown_connections 2024-04-22T01:10:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.821+0000 7f370bb9a640 1 -- 172.21.15.38:0/361909813 wait complete. 2024-04-22T01:10:46.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:46 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4078990631' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:10:46.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:46 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1607569923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:46.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:46 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/701443972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:46.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 -- 172.21.15.38:0/1109563269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100970 msgr2=0x7fd08c100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 --2- 172.21.15.38:0/1109563269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100970 0x7fd08c100dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009900 tx=0x7fd07c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 -- 172.21.15.38:0/1109563269 shutdown_connections 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 --2- 172.21.15.38:0/1109563269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd08c101310 0x7fd08c101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 --2- 172.21.15.38:0/1109563269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100970 0x7fd08c100dd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 --2- 172.21.15.38:0/1109563269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd08c100010 0x7fd08c100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 -- 172.21.15.38:0/1109563269 >> 172.21.15.38:0/1109563269 conn(0x7fd08c0fbc30 msgr2=0x7fd08c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 -- 172.21.15.38:0/1109563269 shutdown_connections 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.936+0000 7fd093e96640 1 -- 172.21.15.38:0/1109563269 wait complete. 2024-04-22T01:10:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 Processor -- start 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 -- start start 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 0x7fd08c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd08c100970 0x7fd08c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd08c101310 0x7fd08c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c10f610 con 0x7fd08c100010 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c10f490 con 0x7fd08c100970 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd093e96640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c10f790 con 0x7fd08c101310 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 0x7fd08c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 0x7fd08c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45528/0 (socket says 172.21.15.38:45528) 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 -- 172.21.15.38:0/1359725436 learned_addr learned my addr 172.21.15.38:0/1359725436 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091431640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd08c100970 0x7fd08c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd092433640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd08c101310 0x7fd08c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 -- 172.21.15.38:0/1359725436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd08c101310 msgr2=0x7fd08c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd08c101310 0x7fd08c1a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 -- 172.21.15.38:0/1359725436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd08c100970 msgr2=0x7fd08c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd08c100970 0x7fd08c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.937+0000 7fd091c32640 1 -- 172.21.15.38:0/1359725436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08c1a1850 con 0x7fd08c100010 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.938+0000 7fd091c32640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 0x7fd08c199c40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c920 tx=0x7fd08800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.938+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088007c70 con 0x7fd08c100010 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.938+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd088007e10 con 0x7fd08c100010 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.938+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08c1a1b40 con 0x7fd08c100010 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.938+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088018890 con 0x7fd08c100010 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.938+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd08c1a2090 con 0x7fd08c100010 2024-04-22T01:10:46.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.939+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd08c10ad60 con 0x7fd08c100010 2024-04-22T01:10:46.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.940+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd088018a70 con 0x7fd08c100010 2024-04-22T01:10:46.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.941+0000 7fd082ffd640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0780a2b90 0x7fd0780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:46.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.941+0000 7fd091431640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0780a2b90 0x7fd0780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:46.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.941+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd0880cf570 con 0x7fd08c100010 2024-04-22T01:10:46.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.941+0000 7fd091431640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0780a2b90 0x7fd0780a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd07c006010 tx=0x7fd07c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:46.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:46.945+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd088014030 con 0x7fd08c100010 2024-04-22T01:10:47.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:46 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4078990631' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:10:47.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:46 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1607569923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:47.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:46 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/701443972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:47.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.129+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd08c02d3b0 con 0x7fd08c100010 2024-04-22T01:10:47.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.130+0000 7fd082ffd640 1 -- 172.21.15.38:0/1359725436 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fd08800de20 con 0x7fd08c100010 2024-04-22T01:10:47.130 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:10:47.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0780a2b90 msgr2=0x7fd0780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0780a2b90 0x7fd0780a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd07c006010 tx=0x7fd07c000bc0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 msgr2=0x7fd08c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 0x7fd08c199c40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c920 tx=0x7fd08800cde0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 shutdown_connections 2024-04-22T01:10:47.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0780a2b90 0x7fd0780a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd08c101310 0x7fd08c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd08c100970 0x7fd08c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 --2- 172.21.15.38:0/1359725436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd08c100010 0x7fd08c199c40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 >> 172.21.15.38:0/1359725436 conn(0x7fd08c0fbc30 msgr2=0x7fd08c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:47.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 shutdown_connections 2024-04-22T01:10:47.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.132+0000 7fd093e96640 1 -- 172.21.15.38:0/1359725436 wait complete. 2024-04-22T01:10:47.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:10:47.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:46 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4078990631' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:10:47.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:46 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1607569923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:47.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:46 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/701443972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 -- 172.21.15.38:0/788204215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 msgr2=0x7efda0109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 --2- 172.21.15.38:0/788204215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda0109bc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd98009ca0 tx=0x7efd9802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 -- 172.21.15.38:0/788204215 shutdown_connections 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 --2- 172.21.15.38:0/788204215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda0109bc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 --2- 172.21.15.38:0/788204215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0102970 0x7efda0102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 --2- 172.21.15.38:0/788204215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0101780 0x7efda0101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 -- 172.21.15.38:0/788204215 >> 172.21.15.38:0/788204215 conn(0x7efda00fccf0 msgr2=0x7efda00ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 -- 172.21.15.38:0/788204215 shutdown_connections 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 -- 172.21.15.38:0/788204215 wait complete. 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 Processor -- start 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.248+0000 7efda580e640 1 -- start start 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0101780 0x7efda0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0102970 0x7efda0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efda010d400 con 0x7efda0103330 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efda010d280 con 0x7efda0102970 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efda010d580 con 0x7efda0101780 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45556/0 (socket says 172.21.15.38:45556) 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 -- 172.21.15.38:0/3213662914 learned_addr learned my addr 172.21.15.38:0/3213662914 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:47.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9e7fc640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0102970 0x7efda0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9effd640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0101780 0x7efda0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 -- 172.21.15.38:0/3213662914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0101780 msgr2=0x7efda0197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0101780 0x7efda0197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 -- 172.21.15.38:0/3213662914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0102970 msgr2=0x7efda0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0102970 0x7efda0197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 -- 172.21.15.38:0/3213662914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda019f660 con 0x7efda0103330 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9effd640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0101780 0x7efda0197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9e7fc640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0102970 0x7efda0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efd9f7fe640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda019eff0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efd98037040 tx=0x7efd98032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98041020 con 0x7efda0103330 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd98034d90 con 0x7efda0103330 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98039040 con 0x7efda0103330 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda019f8f0 con 0x7efda0103330 2024-04-22T01:10:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.249+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda019fde0 con 0x7efda0103330 2024-04-22T01:10:47.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.250+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda0108aa0 con 0x7efda0103330 2024-04-22T01:10:47.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.252+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efd98044020 con 0x7efda0103330 2024-04-22T01:10:47.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.252+0000 7efda480c640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd880a2b90 0x7efd880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.252+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efd980f16b0 con 0x7efda0103330 2024-04-22T01:10:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.252+0000 7efd9effd640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd880a2b90 0x7efd880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.253+0000 7efd9effd640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd880a2b90 0x7efd880a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efda01027d0 tx=0x7efd90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:47.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.256+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efd9804b020 con 0x7efda0103330 2024-04-22T01:10:47.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.440+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efda002d300 con 0x7efda0103330 2024-04-22T01:10:47.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.441+0000 7efda480c640 1 -- 172.21.15.38:0/3213662914 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7efd980333a0 con 0x7efda0103330 2024-04-22T01:10:47.441 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:10:47.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd880a2b90 msgr2=0x7efd880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd880a2b90 0x7efd880a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efda01027d0 tx=0x7efd90005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:47.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 msgr2=0x7efda019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda019eff0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efd98037040 tx=0x7efd98032f20 comp rx=0 tx=0).stop 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 shutdown_connections 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd880a2b90 0x7efd880a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efda0103330 0x7efda019eff0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efda0102970 0x7efda0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 --2- 172.21.15.38:0/3213662914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efda0101780 0x7efda0197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 >> 172.21.15.38:0/3213662914 conn(0x7efda00fccf0 msgr2=0x7efda0105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 shutdown_connections 2024-04-22T01:10:47.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.443+0000 7efda580e640 1 -- 172.21.15.38:0/3213662914 wait complete. 2024-04-22T01:10:47.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:10:47.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- 172.21.15.38:0/3092234795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 msgr2=0x7f5c94101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- 172.21.15.38:0/3092234795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94101d10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c8402bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:47.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- 172.21.15.38:0/3092234795 shutdown_connections 2024-04-22T01:10:47.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- 172.21.15.38:0/3092234795 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 0x7f5c94108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- 172.21.15.38:0/3092234795 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94101d10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- 172.21.15.38:0/3092234795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c941006a0 0x7f5c94100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- 172.21.15.38:0/3092234795 >> 172.21.15.38:0/3092234795 conn(0x7f5c940fbc30 msgr2=0x7f5c940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- 172.21.15.38:0/3092234795 shutdown_connections 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- 172.21.15.38:0/3092234795 wait complete. 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 Processor -- start 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- start start 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c941006a0 0x7f5c94196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 0x7f5c9419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410c300 con 0x7f5c94101890 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410c180 con 0x7f5c941006a0 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.557+0000 7f5c9c279640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410c480 con 0x7f5c94102250 2024-04-22T01:10:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45582/0 (socket says 172.21.15.38:45582) 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 -- 172.21.15.38:0/1050081640 learned_addr learned my addr 172.21.15.38:0/1050081640 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c9a015640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c941006a0 0x7f5c94196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c9a816640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 0x7f5c9419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 -- 172.21.15.38:0/1050081640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 msgr2=0x7f5c9419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 0x7f5c9419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 -- 172.21.15.38:0/1050081640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c941006a0 msgr2=0x7f5c94196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c941006a0 0x7f5c94196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 -- 172.21.15.38:0/1050081640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c94069dc0 con 0x7f5c94101890 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c9a816640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 0x7f5c9419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c99814640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94196e90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009230 tx=0x7f5c84005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84041070 con 0x7f5c94101890 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c840044d0 con 0x7f5c94101890 2024-04-22T01:10:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9406a050 con 0x7f5c94101890 2024-04-22T01:10:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84034440 con 0x7f5c94101890 2024-04-22T01:10:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.558+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9406a540 con 0x7f5c94101890 2024-04-22T01:10:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.559+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c941079c0 con 0x7f5c94101890 2024-04-22T01:10:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.560+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5c84044070 con 0x7f5c94101890 2024-04-22T01:10:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.561+0000 7f5c8b7fe640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5c800a2b90 0x7f5c800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.561+0000 7f5c9a015640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5c800a2b90 0x7f5c800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.561+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5c840f0f60 con 0x7f5c94101890 2024-04-22T01:10:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.562+0000 7f5c9a015640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5c800a2b90 0x7f5c800a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c941016f0 tx=0x7f5c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.565+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5c840bdc20 con 0x7f5c94101890 2024-04-22T01:10:47.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.764+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5c9406a860 con 0x7f5c94101890 2024-04-22T01:10:47.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.764+0000 7f5c8b7fe640 1 -- 172.21.15.38:0/1050081640 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+589 (secure 0 0 0) 0x7f5c840bd520 con 0x7f5c94101890 2024-04-22T01:10:47.764 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3063830265 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3063830265 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2675569647 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2608977615 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3674504753 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3674504753 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/799365618 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3519840670 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/845318699 2024-04-23T01:02:47.985940+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/687303682 2024-04-23T01:03:15.695973+0000 2024-04-22T01:10:47.765 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2024-04-22T01:10:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5c800a2b90 msgr2=0x7f5c800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5c800a2b90 0x7f5c800a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c941016f0 tx=0x7f5c90005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 msgr2=0x7f5c94196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94196e90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009230 tx=0x7f5c84005d50 comp rx=0 tx=0).stop 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 shutdown_connections 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5c800a2b90 0x7f5c800a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c94102250 0x7f5c9419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c94101890 0x7f5c94196e90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 --2- 172.21.15.38:0/1050081640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c941006a0 0x7f5c94196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 >> 172.21.15.38:0/1050081640 conn(0x7f5c940fbc30 msgr2=0x7f5c94104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 shutdown_connections 2024-04-22T01:10:47.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:47.766+0000 7f5c9c279640 1 -- 172.21.15.38:0/1050081640 wait complete. 2024-04-22T01:10:47.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:47 smithi038 ceph-mon[28043]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-22T01:10:47.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:47 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/361909813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:10:47.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:47 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1359725436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:10:47.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:47 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3213662914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:10:47.777 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi038.front.sepia.ceph.com... 2024-04-22T01:10:47.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:47.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-22T01:10:47.800 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[71365]: fuse finished with error 0 and tester_r 0 2024-04-22T01:10:47.802 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:10:48.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:47 smithi203 ceph-mon[31825]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-22T01:10:48.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:47 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/361909813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:10:48.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:47 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1359725436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:10:48.019 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:47 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3213662914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:10:48.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:47 smithi111 ceph-mon[31057]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-22T01:10:48.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:47 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/361909813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:10:48.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:47 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1359725436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:10:48.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:47 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3213662914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:10:49.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:48 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1050081640' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:10:49.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:48 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1050081640' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:10:49.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:48 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1050081640' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:10:50.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:49 smithi203 ceph-mon[31825]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-22T01:10:50.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:49 smithi111 ceph-mon[31057]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-22T01:10:50.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:49 smithi038 ceph-mon[28043]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-22T01:10:51.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:50 smithi111 ceph-mon[31057]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-22T01:10:51.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:50 smithi038 ceph-mon[28043]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-22T01:10:51.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:50 smithi203 ceph-mon[31825]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-22T01:10:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:52 smithi111 ceph-mon[31057]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:53.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:52 smithi038 ceph-mon[28043]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:53.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:52 smithi203 ceph-mon[31825]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:53.904 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:10:53.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:10:53.904 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-22T01:10:53.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.081+0000 7f01fada2640 1 -- 172.21.15.38:0/597775624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 msgr2=0x7f01f4102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.081+0000 7f01fada2640 1 --2- 172.21.15.38:0/597775624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f4102260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009900 tx=0x7f01f002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- 172.21.15.38:0/597775624 shutdown_connections 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 --2- 172.21.15.38:0/597775624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01f4105150 0x7f01f41055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 --2- 172.21.15.38:0/597775624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f41027a0 0x7f01f4104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 --2- 172.21.15.38:0/597775624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f4102260 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- 172.21.15.38:0/597775624 >> 172.21.15.38:0/597775624 conn(0x7f01f40fba50 msgr2=0x7f01f40fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- 172.21.15.38:0/597775624 shutdown_connections 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- 172.21.15.38:0/597775624 wait complete. 2024-04-22T01:10:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 Processor -- start 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- start start 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01f41027a0 0x7f01f419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f4105150 0x7f01f41a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4113880 con 0x7f01f40ffe30 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4113700 con 0x7f01f4105150 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.082+0000 7f01fada2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4113a00 con 0x7f01f41027a0 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45608/0 (socket says 172.21.15.38:45608) 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 -- 172.21.15.38:0/2994413679 learned_addr learned my addr 172.21.15.38:0/2994413679 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f959f640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01f41027a0 0x7f01f419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01fa5a1640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f4105150 0x7f01f41a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 -- 172.21.15.38:0/2994413679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01f41027a0 msgr2=0x7f01f419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01f41027a0 0x7f01f419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 -- 172.21.15.38:0/2994413679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f4105150 msgr2=0x7f01f41a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f4105150 0x7f01f41a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 -- 172.21.15.38:0/2994413679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f41a5ad0 con 0x7f01f40ffe30 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01fa5a1640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f4105150 0x7f01f41a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01f9da0640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f419deb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f01f0000c00 tx=0x7f01f0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0043020 con 0x7f01f40ffe30 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f41a5d60 con 0x7f01f40ffe30 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f00054e0 con 0x7f01f40ffe30 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0003e60 con 0x7f01f40ffe30 2024-04-22T01:10:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.083+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f41a61f0 con 0x7f01f40ffe30 2024-04-22T01:10:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.084+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f40fd540 con 0x7f01f40ffe30 2024-04-22T01:10:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.085+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f01f0046020 con 0x7f01f40ffe30 2024-04-22T01:10:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.086+0000 7f01eaffd640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f01e00a2b90 0x7f01e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.086+0000 7f01f959f640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f01e00a2b90 0x7f01e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.086+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f01f00f1950 con 0x7f01f40ffe30 2024-04-22T01:10:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.087+0000 7f01f959f640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f01e00a2b90 0x7f01e00a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f01e4005d00 tx=0x7f01e4005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.090+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f01f004d020 con 0x7f01f40ffe30 2024-04-22T01:10:54.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.277+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01f40623a0 con 0x7f01f40ffe30 2024-04-22T01:10:54.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.279+0000 7f01eaffd640 1 -- 172.21.15.38:0/2994413679 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f01f00be670 con 0x7f01f40ffe30 2024-04-22T01:10:54.279 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:54.279 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f01e00a2b90 msgr2=0x7f01e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f01e00a2b90 0x7f01e00a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f01e4005d00 tx=0x7f01e4005c50 comp rx=0 tx=0).stop 2024-04-22T01:10:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 msgr2=0x7f01f419deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f419deb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f01f0000c00 tx=0x7f01f0005dd0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 shutdown_connections 2024-04-22T01:10:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f01e00a2b90 0x7f01e00a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f4105150 0x7f01f41a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01f41027a0 0x7f01f419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 --2- 172.21.15.38:0/2994413679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f01f40ffe30 0x7f01f419deb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 >> 172.21.15.38:0/2994413679 conn(0x7f01f40fba50 msgr2=0x7f01f4103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:54.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 shutdown_connections 2024-04-22T01:10:54.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.281+0000 7f01fada2640 1 -- 172.21.15.38:0/2994413679 wait complete. 2024-04-22T01:10:54.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:10:54.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.393+0000 7fa4b5696640 1 -- 172.21.15.38:0/2874686402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102250 msgr2=0x7fa4b0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.393+0000 7fa4b5696640 1 --2- 172.21.15.38:0/2874686402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102250 0x7fa4b0108ae0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009ca0 tx=0x7fa4a802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 -- 172.21.15.38:0/2874686402 shutdown_connections 2024-04-22T01:10:54.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 --2- 172.21.15.38:0/2874686402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102250 0x7fa4b0108ae0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 --2- 172.21.15.38:0/2874686402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0101890 0x7fa4b0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 --2- 172.21.15.38:0/2874686402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b01006a0 0x7fa4b0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 -- 172.21.15.38:0/2874686402 >> 172.21.15.38:0/2874686402 conn(0x7fa4b00fbc30 msgr2=0x7fa4b00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 -- 172.21.15.38:0/2874686402 shutdown_connections 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 -- 172.21.15.38:0/2874686402 wait complete. 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 Processor -- start 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 -- start start 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 0x7fa4b006a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.394+0000 7fa4b5696640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101890 0x7fa4b019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4b5696640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0102250 0x7fa4b019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4b5696640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b010c320 con 0x7fa4b01006a0 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4b5696640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b010c1a0 con 0x7fa4b0102250 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4b5696640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b010c4a0 con 0x7fa4b0101890 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 0x7fa4b006a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 0x7fa4b006a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45622/0 (socket says 172.21.15.38:45622) 2024-04-22T01:10:54.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 -- 172.21.15.38:0/3181883930 learned_addr learned my addr 172.21.15.38:0/3181883930 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4af7fe640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0102250 0x7fa4b019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4ae7fc640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101890 0x7fa4b019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 -- 172.21.15.38:0/3181883930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101890 msgr2=0x7fa4b019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101890 0x7fa4b019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 -- 172.21.15.38:0/3181883930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0102250 msgr2=0x7fa4b019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0102250 0x7fa4b019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 -- 172.21.15.38:0/3181883930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b019f620 con 0x7fa4b01006a0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4ae7fc640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101890 0x7fa4b019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4aeffd640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 0x7fa4b006a970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000c970 tx=0x7fa4a000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0004220 con 0x7fa4b01006a0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a00043c0 con 0x7fa4b01006a0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b019f910 con 0x7fa4b01006a0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0004a00 con 0x7fa4b01006a0 2024-04-22T01:10:54.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.395+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b019fe10 con 0x7fa4b01006a0 2024-04-22T01:10:54.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.396+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b0107960 con 0x7fa4b01006a0 2024-04-22T01:10:54.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.397+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa4a00079b0 con 0x7fa4b01006a0 2024-04-22T01:10:54.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.398+0000 7fa497fff640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4980a2b90 0x7fa4980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.398+0000 7fa4ae7fc640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4980a2b90 0x7fa4980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.398+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa4a00cf040 con 0x7fa4b01006a0 2024-04-22T01:10:54.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.398+0000 7fa4ae7fc640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4980a2b90 0x7fa4980a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa4b006a6e0 tx=0x7fa4a400a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:54.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.402+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa4a0014030 con 0x7fa4b01006a0 2024-04-22T01:10:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.592+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4b002d170 con 0x7fa4b01006a0 2024-04-22T01:10:54.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.594+0000 7fa497fff640 1 -- 172.21.15.38:0/3181883930 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7fa4a009bd60 con 0x7fa4b01006a0 2024-04-22T01:10:54.594 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:10:54.595 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:10:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4980a2b90 msgr2=0x7fa4980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4980a2b90 0x7fa4980a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa4b006a6e0 tx=0x7fa4a400a5c0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 msgr2=0x7fa4b006a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 0x7fa4b006a970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000c970 tx=0x7fa4a000ce30 comp rx=0 tx=0).stop 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 shutdown_connections 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4980a2b90 0x7fa4980a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0102250 0x7fa4b019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101890 0x7fa4b019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 --2- 172.21.15.38:0/3181883930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b01006a0 0x7fa4b006a970 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 >> 172.21.15.38:0/3181883930 conn(0x7fa4b00fbc30 msgr2=0x7fa4b0104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 shutdown_connections 2024-04-22T01:10:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.596+0000 7fa4b5696640 1 -- 172.21.15.38:0/3181883930 wait complete. 2024-04-22T01:10:54.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 -- 172.21.15.38:0/2170692816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 msgr2=0x7f0664106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 --2- 172.21.15.38:0/2170692816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664106fa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f064c009900 tx=0x7f064c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 -- 172.21.15.38:0/2170692816 shutdown_connections 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 --2- 172.21.15.38:0/2170692816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 0x7f06641098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 --2- 172.21.15.38:0/2170692816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664106fa0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 --2- 172.21.15.38:0/2170692816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0664102280 0x7f0664104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 -- 172.21.15.38:0/2170692816 >> 172.21.15.38:0/2170692816 conn(0x7f06640fbc30 msgr2=0x7f06640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 -- 172.21.15.38:0/2170692816 shutdown_connections 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.712+0000 7f0668c27640 1 -- 172.21.15.38:0/2170692816 wait complete. 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 Processor -- start 2024-04-22T01:10:54.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 -- start start 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0664102280 0x7f0664197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 0x7f066419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f066410d3f0 con 0x7f0664104bb0 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f066410d270 con 0x7f0664102280 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0668c27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f066410d570 con 0x7f06641074e0 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0661d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0661d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45640/0 (socket says 172.21.15.38:45640) 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f0661d9b640 1 -- 172.21.15.38:0/614275124 learned_addr learned my addr 172.21.15.38:0/614275124 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.713+0000 7f066259c640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0664102280 0x7f0664197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0662d9d640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 0x7f066419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0661d9b640 1 -- 172.21.15.38:0/614275124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 msgr2=0x7f066419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0661d9b640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 0x7f066419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0661d9b640 1 -- 172.21.15.38:0/614275124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0664102280 msgr2=0x7f0664197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0661d9b640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0664102280 0x7f0664197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0661d9b640 1 -- 172.21.15.38:0/614275124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066419f6c0 con 0x7f0664104bb0 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0662d9d640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 0x7f066419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0661d9b640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664197fe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f064c005ab0 tx=0x7f064c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c041070 con 0x7f0664104bb0 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066419f950 con 0x7f0664104bb0 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f064c0054e0 con 0x7f0664104bb0 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c034440 con 0x7f0664104bb0 2024-04-22T01:10:54.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.714+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066419fe90 con 0x7f0664104bb0 2024-04-22T01:10:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.715+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06640fd720 con 0x7f0664104bb0 2024-04-22T01:10:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.716+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f064c044070 con 0x7f0664104bb0 2024-04-22T01:10:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.717+0000 7f065b7fe640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06500a2b90 0x7f06500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.717+0000 7f066259c640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06500a2b90 0x7f06500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.717+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f064c0f1180 con 0x7f0664104bb0 2024-04-22T01:10:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.717+0000 7f066259c640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06500a2b90 0x7f06500a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f065c005fd0 tx=0x7f065c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:54.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.720+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f064c0bde40 con 0x7f0664104bb0 2024-04-22T01:10:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:54 smithi038 ceph-mon[28043]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2994413679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3181883930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:54.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:54.944+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f066402d390 con 0x7f0664104bb0 2024-04-22T01:10:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:54 smithi111 ceph-mon[31057]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2994413679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3181883930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:54 smithi203 ceph-mon[31825]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2994413679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:55.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3181883930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:10:55.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.868+0000 7f065b7fe640 1 -- 172.21.15.38:0/614275124 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f064c0bd740 con 0x7f0664104bb0 2024-04-22T01:10:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06500a2b90 msgr2=0x7f06500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06500a2b90 0x7f06500a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f065c005fd0 tx=0x7f065c005850 comp rx=0 tx=0).stop 2024-04-22T01:10:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 msgr2=0x7f0664197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664197fe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f064c005ab0 tx=0x7f064c005dc0 comp rx=0 tx=0).stop 2024-04-22T01:10:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 shutdown_connections 2024-04-22T01:10:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06500a2b90 0x7f06500a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:55.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06641074e0 0x7f066419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:55.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0664104bb0 0x7f0664197fe0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:55.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 --2- 172.21.15.38:0/614275124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0664102280 0x7f0664197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:55.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 >> 172.21.15.38:0/614275124 conn(0x7f06640fbc30 msgr2=0x7f0664102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:55.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 shutdown_connections 2024-04-22T01:10:55.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.871+0000 7f0668c27640 1 -- 172.21.15.38:0/614275124 wait complete. 2024-04-22T01:10:55.884 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-22T01:10:55.884 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 -- 172.21.15.38:0/539141201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0102220 msgr2=0x7f20e0104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/539141201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0102220 0x7f20e0104610 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009900 tx=0x7f20d002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 -- 172.21.15.38:0/539141201 shutdown_connections 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/539141201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0107480 0x7f20e01098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/539141201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e0104b50 0x7f20e0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/539141201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0102220 0x7f20e0104610 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 -- 172.21.15.38:0/539141201 >> 172.21.15.38:0/539141201 conn(0x7f20e00fbc30 msgr2=0x7f20e00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 -- 172.21.15.38:0/539141201 shutdown_connections 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.996+0000 7f20e5b1e640 1 -- 172.21.15.38:0/539141201 wait complete. 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 Processor -- start 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 -- start start 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e0102220 0x7f20e0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 0x7f20e0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0107480 0x7f20e019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d450 con 0x7f20e0104b50 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d2d0 con 0x7f20e0107480 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20e5b1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d5d0 con 0x7f20e0102220 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20deffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 0x7f20e0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:56.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20deffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 0x7f20e0198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45644/0 (socket says 172.21.15.38:45644) 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20deffd640 1 -- 172.21.15.38:0/2865106013 learned_addr learned my addr 172.21.15.38:0/2865106013 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.997+0000 7f20df7fe640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e0102220 0x7f20e0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20deffd640 1 -- 172.21.15.38:0/2865106013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e0102220 msgr2=0x7f20e0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20deffd640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e0102220 0x7f20e0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20deffd640 1 -- 172.21.15.38:0/2865106013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0107480 msgr2=0x7f20e019f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20dffff640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0107480 0x7f20e019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20deffd640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0107480 0x7f20e019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20deffd640 1 -- 172.21.15.38:0/2865106013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e019f740 con 0x7f20e0104b50 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20deffd640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 0x7f20e0198060 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f20d400e9f0 tx=0x7f20d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:56.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d400ccc0 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e019fa30 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d4005500 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4010690 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.998+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e019ff30 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:55.999+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e00fd720 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.000+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f20d40046c0 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.001+0000 7f20dcff9640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f20c80a2b90 0x7f20c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.001+0000 7f20df7fe640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f20c80a2b90 0x7f20c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.001+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f20d40cf7b0 con 0x7f20e0104b50 2024-04-22T01:10:56.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.001+0000 7f20df7fe640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f20c80a2b90 0x7f20c80a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f20d0006010 tx=0x7f20d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.005+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f20d409c470 con 0x7f20e0104b50 2024-04-22T01:10:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:55 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/614275124' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T01:10:56.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:55 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/614275124' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T01:10:56.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.229+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f20e0062370 con 0x7f20e0104b50 2024-04-22T01:10:56.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:55 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/614275124' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T01:10:56.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.869+0000 7f20dcff9640 1 -- 172.21.15.38:0/2865106013 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f20d409bd70 con 0x7f20e0104b50 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f20c80a2b90 msgr2=0x7f20c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f20c80a2b90 0x7f20c80a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f20d0006010 tx=0x7f20d0000bc0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 msgr2=0x7f20e0198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 0x7f20e0198060 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f20d400e9f0 tx=0x7f20d400eeb0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 shutdown_connections 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f20c80a2b90 0x7f20c80a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e0107480 0x7f20e019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e0104b50 0x7f20e0198060 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 --2- 172.21.15.38:0/2865106013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e0102220 0x7f20e0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:56.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.873+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 >> 172.21.15.38:0/2865106013 conn(0x7f20e00fbc30 msgr2=0x7f20e0102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:56.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.874+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 shutdown_connections 2024-04-22T01:10:56.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:56.874+0000 7f20e5b1e640 1 -- 172.21.15.38:0/2865106013 wait complete. 2024-04-22T01:10:56.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 -- 172.21.15.38:0/1110824964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 msgr2=0x7f8618109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 --2- 172.21.15.38:0/1110824964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618109be0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8610009ca0 tx=0x7f861002c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 -- 172.21.15.38:0/1110824964 shutdown_connections 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 --2- 172.21.15.38:0/1110824964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618109be0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 --2- 172.21.15.38:0/1110824964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8618102990 0x7f8618102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 --2- 172.21.15.38:0/1110824964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86181017a0 0x7f8618101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 -- 172.21.15.38:0/1110824964 >> 172.21.15.38:0/1110824964 conn(0x7f86180fcd10 msgr2=0x7f86180ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 -- 172.21.15.38:0/1110824964 shutdown_connections 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 -- 172.21.15.38:0/1110824964 wait complete. 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 Processor -- start 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.001+0000 7f861fc88640 1 -- start start 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861fc88640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86181017a0 0x7f86180754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861fc88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8618102990 0x7f8618073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861fc88640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861fc88640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f861810d400 con 0x7f8618103350 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861fc88640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f861810d280 con 0x7f86181017a0 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861fc88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f861810d580 con 0x7f8618102990 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44316/0 (socket says 172.21.15.38:44316) 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 -- 172.21.15.38:0/3299112408 learned_addr learned my addr 172.21.15.38:0/3299112408 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861da24640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86181017a0 0x7f86180754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861d223640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8618102990 0x7f8618073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 -- 172.21.15.38:0/3299112408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8618102990 msgr2=0x7f8618073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8618102990 0x7f8618073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 -- 172.21.15.38:0/3299112408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86181017a0 msgr2=0x7f86180754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86181017a0 0x7f86180754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 -- 172.21.15.38:0/3299112408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8618074900 con 0x7f8618103350 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f861e225640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618074070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8610032530 tx=0x7f8610035b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.002+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8610039040 con 0x7f8618103350 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.003+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8610044070 con 0x7f8618103350 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.003+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86181a1ae0 con 0x7f8618103350 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.003+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8610004430 con 0x7f8618103350 2024-04-22T01:10:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.003+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86181a2030 con 0x7f8618103350 2024-04-22T01:10:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.004+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8618108a60 con 0x7f8618103350 2024-04-22T01:10:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.005+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f86100045d0 con 0x7f8618103350 2024-04-22T01:10:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.005+0000 7f860effd640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86040a2b90 0x7f86040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.005+0000 7f861da24640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86040a2b90 0x7f86040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.005+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f86100f1820 con 0x7f8618103350 2024-04-22T01:10:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.006+0000 7f861da24640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86040a2b90 0x7f86040a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f86181027f0 tx=0x7f8614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.010+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f86100be540 con 0x7f8618103350 2024-04-22T01:10:57.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:56 smithi111 ceph-mon[31057]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:57.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/614275124' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T01:10:57.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:56 smithi111 ceph-mon[31057]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:57.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2865106013' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T01:10:57.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:56 smithi038 ceph-mon[28043]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/614275124' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T01:10:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:56 smithi038 ceph-mon[28043]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:57.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2865106013' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T01:10:57.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:56 smithi203 ceph-mon[31825]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:57.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/614275124' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T01:10:57.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:56 smithi203 ceph-mon[31825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:57.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2865106013' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T01:10:57.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.233+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f8618062340 con 0x7f8618103350 2024-04-22T01:10:57.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.885+0000 7f860effd640 1 -- 172.21.15.38:0/3299112408 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f86100098b0 con 0x7f8618103350 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86040a2b90 msgr2=0x7f86040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86040a2b90 0x7f86040a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f86181027f0 tx=0x7f8614005f90 comp rx=0 tx=0).stop 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 msgr2=0x7f8618074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618074070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8610032530 tx=0x7f8610035b60 comp rx=0 tx=0).stop 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 shutdown_connections 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86040a2b90 0x7f86040a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8618103350 0x7f8618074070 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8618102990 0x7f8618073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 --2- 172.21.15.38:0/3299112408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86181017a0 0x7f86180754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:57.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 >> 172.21.15.38:0/3299112408 conn(0x7f86180fcd10 msgr2=0x7f8618105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:57.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 shutdown_connections 2024-04-22T01:10:57.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:57.888+0000 7f861fc88640 1 -- 172.21.15.38:0/3299112408 wait complete. 2024-04-22T01:10:57.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 -- 172.21.15.38:0/687795300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 msgr2=0x7f1e80106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 --2- 172.21.15.38:0/687795300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80106f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009900 tx=0x7f1e7002bc30 comp rx=0 tx=0).stop 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 -- 172.21.15.38:0/687795300 shutdown_connections 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 --2- 172.21.15.38:0/687795300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80107480 0x7f1e801098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 --2- 172.21.15.38:0/687795300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80106f40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 --2- 172.21.15.38:0/687795300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80102220 0x7f1e80104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 -- 172.21.15.38:0/687795300 >> 172.21.15.38:0/687795300 conn(0x7f1e800fbc30 msgr2=0x7f1e800fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 -- 172.21.15.38:0/687795300 shutdown_connections 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.017+0000 7f1e88713640 1 -- 172.21.15.38:0/687795300 wait complete. 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 Processor -- start 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 -- start start 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 0x7f1e80197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80107480 0x7f1e8019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8010d430 con 0x7f1e80104b50 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8010d2b0 con 0x7f1e80102220 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e88713640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8010d5b0 con 0x7f1e80107480 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e864af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 0x7f1e80197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e85cae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e864af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 0x7f1e80197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51556/0 (socket says 172.21.15.38:51556) 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.018+0000 7f1e864af640 1 -- 172.21.15.38:0/3232229654 learned_addr learned my addr 172.21.15.38:0/3232229654 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e86cb0640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80107480 0x7f1e8019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e85cae640 1 -- 172.21.15.38:0/3232229654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80107480 msgr2=0x7f1e8019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e85cae640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80107480 0x7f1e8019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e85cae640 1 -- 172.21.15.38:0/3232229654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 msgr2=0x7f1e80197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e85cae640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 0x7f1e80197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e85cae640 1 -- 172.21.15.38:0/3232229654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8019f700 con 0x7f1e80104b50 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e864af640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 0x7f1e80197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e86cb0640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80107480 0x7f1e8019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e85cae640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80198020 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1e7002c170 tx=0x7f1e7002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70041070 con 0x7f1e80104b50 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e70004450 con 0x7f1e80104b50 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e8019f990 con 0x7f1e80104b50 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70034440 con 0x7f1e80104b50 2024-04-22T01:10:58.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.019+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e8019fea0 con 0x7f1e80104b50 2024-04-22T01:10:58.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.020+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e800fd720 con 0x7f1e80104b50 2024-04-22T01:10:58.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.021+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1e70044070 con 0x7f1e80104b50 2024-04-22T01:10:58.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.022+0000 7f1e777fe640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e6c0a2c10 0x7f1e6c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:58.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.022+0000 7f1e864af640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e6c0a2c10 0x7f1e6c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:58.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.022+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1e700f0fe0 con 0x7f1e80104b50 2024-04-22T01:10:58.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.022+0000 7f1e864af640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e6c0a2c10 0x7f1e6c0a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0104e0 tx=0x7f1e7c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:58.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.026+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e700bdd00 con 0x7f1e80104b50 2024-04-22T01:10:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T01:10:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2865106013' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T01:10:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T01:10:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-22T01:10:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T01:10:58.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:57 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3299112408' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T01:10:58.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T01:10:58.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2865106013' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T01:10:58.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T01:10:58.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:58.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-22T01:10:58.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T01:10:58.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:57 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3299112408' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T01:10:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.252+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f1e8002d170 con 0x7f1e80104b50 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2865106013' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T01:10:58.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:57 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3299112408' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T01:10:58.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.894+0000 7f1e777fe640 1 -- 172.21.15.38:0/3232229654 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f1e700bd600 con 0x7f1e80104b50 2024-04-22T01:10:58.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e6c0a2c10 msgr2=0x7f1e6c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:58.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e6c0a2c10 0x7f1e6c0a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0104e0 tx=0x7f1e7c00b410 comp rx=0 tx=0).stop 2024-04-22T01:10:58.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 msgr2=0x7f1e80198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:58.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80198020 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1e7002c170 tx=0x7f1e7002c6c0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 shutdown_connections 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e6c0a2c10 0x7f1e6c0a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e80107480 0x7f1e8019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e80104b50 0x7f1e80198020 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 --2- 172.21.15.38:0/3232229654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e80102220 0x7f1e80197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 >> 172.21.15.38:0/3232229654 conn(0x7f1e800fbc30 msgr2=0x7f1e80102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 shutdown_connections 2024-04-22T01:10:58.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:58.898+0000 7f1e88713640 1 -- 172.21.15.38:0/3232229654 wait complete. 2024-04-22T01:10:58.915 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-22T01:10:58.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-22T01:10:59.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 -- 172.21.15.38:0/1378615754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd310102250 msgr2=0x7fd310108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 --2- 172.21.15.38:0/1378615754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd310102250 0x7fd310108ae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd308009ca0 tx=0x7fd30802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 -- 172.21.15.38:0/1378615754 shutdown_connections 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 --2- 172.21.15.38:0/1378615754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd310102250 0x7fd310108ae0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 --2- 172.21.15.38:0/1378615754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd310101890 0x7fd310101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 --2- 172.21.15.38:0/1378615754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3101006a0 0x7fd310100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 -- 172.21.15.38:0/1378615754 >> 172.21.15.38:0/1378615754 conn(0x7fd3100fbc30 msgr2=0x7fd3100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 -- 172.21.15.38:0/1378615754 shutdown_connections 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.026+0000 7fd316ac9640 1 -- 172.21.15.38:0/1378615754 wait complete. 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 Processor -- start 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 -- start start 2024-04-22T01:10:59.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 0x7fd310197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd310101890 0x7fd310197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd310102250 0x7fd31019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd31010c320 con 0x7fd3101006a0 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd31010c1a0 con 0x7fd310101890 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd316ac9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd31010c4a0 con 0x7fd310102250 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd314865640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 0x7fd310197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd314865640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 0x7fd310197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44346/0 (socket says 172.21.15.38:44346) 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd314865640 1 -- 172.21.15.38:0/919693044 learned_addr learned my addr 172.21.15.38:0/919693044 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd30ffff640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd310101890 0x7fd310197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd315066640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd310102250 0x7fd31019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.027+0000 7fd314865640 1 -- 172.21.15.38:0/919693044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd310102250 msgr2=0x7fd31019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd314865640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd310102250 0x7fd31019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd314865640 1 -- 172.21.15.38:0/919693044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd310101890 msgr2=0x7fd310197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd314865640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd310101890 0x7fd310197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd314865640 1 -- 172.21.15.38:0/919693044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31019f6a0 con 0x7fd3101006a0 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd314865640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 0x7fd310197a80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd30000e9c0 tx=0x7fd30000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30000cd80 con 0x7fd3101006a0 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3000035f0 con 0x7fd3101006a0 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31019f990 con 0x7fd3101006a0 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300010690 con 0x7fd3101006a0 2024-04-22T01:10:59.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.028+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31019fee0 con 0x7fd3101006a0 2024-04-22T01:10:59.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.029+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3101a0170 con 0x7fd3101006a0 2024-04-22T01:10:59.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.030+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd3000037a0 con 0x7fd3101006a0 2024-04-22T01:10:59.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.031+0000 7fd30dffb640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd2fc0a2b90 0x7fd2fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:10:59.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.031+0000 7fd30ffff640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd2fc0a2b90 0x7fd2fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:10:59.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.031+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd3000cf770 con 0x7fd3101006a0 2024-04-22T01:10:59.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.031+0000 7fd30ffff640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd2fc0a2b90 0x7fd2fc0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd310198f90 tx=0x7fd2f800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:10:59.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.036+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd30009c430 con 0x7fd3101006a0 2024-04-22T01:10:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3299112408' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T01:10:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] up:active 2024-04-22T01:10:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T01:10:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-22T01:10:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-22T01:10:59.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T01:10:59.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:58 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3232229654' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T01:10:59.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3299112408' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] up:active 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T01:10:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:58 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3232229654' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T01:10:59.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.245+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd3101a0340 con 0x7fd2fc0a2b90 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3299112408' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] up:active 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-22T01:10:59.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T01:10:59.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:58 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3232229654' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T01:10:59.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:10:59.250+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:10:59.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:10:59.250+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:10:59.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:10:59.250+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:10:59.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:10:59.250+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:10:59.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:10:59.250+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:10:59.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.935+0000 7fd30dffb640 1 -- 172.21.15.38:0/919693044 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd3101a0340 con 0x7fd2fc0a2b90 2024-04-22T01:10:59.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd2fc0a2b90 msgr2=0x7fd2fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:59.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd2fc0a2b90 0x7fd2fc0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd310198f90 tx=0x7fd2f800b390 comp rx=0 tx=0).stop 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 msgr2=0x7fd310197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 0x7fd310197a80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd30000e9c0 tx=0x7fd30000ee80 comp rx=0 tx=0).stop 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 shutdown_connections 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd2fc0a2b90 0x7fd2fc0a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd310102250 0x7fd31019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd310101890 0x7fd310197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 --2- 172.21.15.38:0/919693044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3101006a0 0x7fd310197a80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 >> 172.21.15.38:0/919693044 conn(0x7fd3100fbc30 msgr2=0x7fd3100fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 shutdown_connections 2024-04-22T01:10:59.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:10:59.938+0000 7fd316ac9640 1 -- 172.21.15.38:0/919693044 wait complete. 2024-04-22T01:10:59.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 -- 172.21.15.38:0/2029894735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944101890 msgr2=0x7f3944101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2029894735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944101890 0x7f3944101d10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3938009900 tx=0x7f393802bc30 comp rx=0 tx=0).stop 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 -- 172.21.15.38:0/2029894735 shutdown_connections 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2029894735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944102250 0x7f3944108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2029894735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944101890 0x7f3944101d10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2029894735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 0x7f3944100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 -- 172.21.15.38:0/2029894735 >> 172.21.15.38:0/2029894735 conn(0x7f39440fbc30 msgr2=0x7f39440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:00.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 -- 172.21.15.38:0/2029894735 shutdown_connections 2024-04-22T01:11:00.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.064+0000 7f3949a8b640 1 -- 172.21.15.38:0/2029894735 wait complete. 2024-04-22T01:11:00.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 Processor -- start 2024-04-22T01:11:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 -- start start 2024-04-22T01:11:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 0x7f394406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944101890 0x7f394419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 0x7f394419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f394410c390 con 0x7f3944102250 2024-04-22T01:11:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f394410c210 con 0x7f3944101890 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3949a8b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f394410c510 con 0x7f39441006a0 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3943fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 0x7f394419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.065+0000 7f3942ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944101890 0x7f394419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 0x7f394419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44380/0 (socket says 172.21.15.38:44380) 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 -- 172.21.15.38:0/2040148235 learned_addr learned my addr 172.21.15.38:0/2040148235 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f39437fe640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 0x7f394406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 -- 172.21.15.38:0/2040148235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 msgr2=0x7f394406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 0x7f394406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 -- 172.21.15.38:0/2040148235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944101890 msgr2=0x7f394419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944101890 0x7f394419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 -- 172.21.15.38:0/2040148235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394419f7c0 con 0x7f3944102250 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3942ffd640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944101890 0x7f394419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f39437fe640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 0x7f394406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:00.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3943fff640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 0x7f394419eeb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f393c00e970 tx=0x7f393c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:00.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c00ccc0 con 0x7f3944102250 2024-04-22T01:11:00.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f393c00ce60 con 0x7f3944102250 2024-04-22T01:11:00.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f394419fa50 con 0x7f3944102250 2024-04-22T01:11:00.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c010490 con 0x7f3944102250 2024-04-22T01:11:00.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.066+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f394419ff70 con 0x7f3944102250 2024-04-22T01:11:00.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.067+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39441079c0 con 0x7f3944102250 2024-04-22T01:11:00.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.068+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f393c005020 con 0x7f3944102250 2024-04-22T01:11:00.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.069+0000 7f3940ff9640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f392c0a2c10 0x7f392c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.069+0000 7f39437fe640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f392c0a2c10 0x7f392c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.069+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f393c0cf750 con 0x7f3944102250 2024-04-22T01:11:00.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.069+0000 7f39437fe640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f392c0a2c10 0x7f392c0a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f39441016f0 tx=0x7f3934009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:00.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.073+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f393c09c470 con 0x7f3944102250 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3232229654' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] up:active 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-22T01:11:00.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:10:59 smithi038 ceph-mon[28043]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3232229654' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] up:active 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-22T01:11:00.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:10:59 smithi203 ceph-mon[31825]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:00.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.286+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3944061c10 con 0x7f392c0a2c10 2024-04-22T01:11:00.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.311+0000 7f3940ff9640 1 -- 172.21.15.38:0/2040148235 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3944061c10 con 0x7f392c0a2c10 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f392c0a2c10 msgr2=0x7f392c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f392c0a2c10 0x7f392c0a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f39441016f0 tx=0x7f3934009570 comp rx=0 tx=0).stop 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 msgr2=0x7f394419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 0x7f394419eeb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f393c00e970 tx=0x7f393c00ed70 comp rx=0 tx=0).stop 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 shutdown_connections 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f392c0a2c10 0x7f392c0a50d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3944102250 0x7f394419eeb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3944101890 0x7f394419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 --2- 172.21.15.38:0/2040148235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39441006a0 0x7f394406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.313+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 >> 172.21.15.38:0/2040148235 conn(0x7f39440fbc30 msgr2=0x7f3944104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:00.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.314+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 shutdown_connections 2024-04-22T01:11:00.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.314+0000 7f3949a8b640 1 -- 172.21.15.38:0/2040148235 wait complete. 2024-04-22T01:11:00.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3232229654' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] up:active 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-22T01:11:00.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:10:59 smithi111 ceph-mon[31057]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 -- 172.21.15.38:0/2383449738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584101790 msgr2=0x7f3584101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 --2- 172.21.15.38:0/2383449738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584101790 0x7f3584101bb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3580009900 tx=0x7f358002bc30 comp rx=0 tx=0).stop 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 -- 172.21.15.38:0/2383449738 shutdown_connections 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 --2- 172.21.15.38:0/2383449738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584103340 0x7f3584109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 --2- 172.21.15.38:0/2383449738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3584102980 0x7f3584102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 --2- 172.21.15.38:0/2383449738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584101790 0x7f3584101bb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 -- 172.21.15.38:0/2383449738 >> 172.21.15.38:0/2383449738 conn(0x7f35840fcd20 msgr2=0x7f35840ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.434+0000 7f358c3a0640 1 -- 172.21.15.38:0/2383449738 shutdown_connections 2024-04-22T01:11:00.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.435+0000 7f358c3a0640 1 -- 172.21.15.38:0/2383449738 wait complete. 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.435+0000 7f358c3a0640 1 Processor -- start 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.435+0000 7f358c3a0640 1 -- start start 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358c3a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 0x7f3584197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358c3a0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 0x7f3584198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358c3a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3584103340 0x7f358419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 0x7f3584198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358a13c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 0x7f3584197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358a13c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 0x7f3584197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51634/0 (socket says 172.21.15.38:51634) 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 0x7f3584198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44406/0 (socket says 172.21.15.38:44406) 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358a13c640 1 -- 172.21.15.38:0/1419710701 learned_addr learned my addr 172.21.15.38:0/1419710701 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f358410d400 con 0x7f3584102980 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f358410d280 con 0x7f3584101790 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f358410d580 con 0x7f3584103340 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358a93d640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3584103340 0x7f358419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 -- 172.21.15.38:0/1419710701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3584103340 msgr2=0x7f358419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3584103340 0x7f358419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 -- 172.21.15.38:0/1419710701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 msgr2=0x7f3584197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 0x7f3584197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358993b640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358419f720 con 0x7f3584102980 2024-04-22T01:11:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.436+0000 7f358a13c640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 0x7f3584197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.437+0000 7f358993b640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 0x7f3584198040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f357400e9f0 tx=0x7f357400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.437+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357400ccc0 con 0x7f3584102980 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.437+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f358419fa10 con 0x7f3584102980 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.437+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f358419ff60 con 0x7f3584102980 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.437+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3574005020 con 0x7f3584102980 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.437+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35740049e0 con 0x7f3584102980 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.438+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35680052c0 con 0x7f3584102980 2024-04-22T01:11:00.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.439+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3574004cc0 con 0x7f3584102980 2024-04-22T01:11:00.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.440+0000 7f357b7fe640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35700a2c10 0x7f35700a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.440+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f35740cf0f0 con 0x7f3584102980 2024-04-22T01:11:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.440+0000 7f358a13c640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35700a2c10 0x7f35700a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.441+0000 7f358a13c640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35700a2c10 0x7f35700a50d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f358002c8e0 tx=0x7f358002c670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:00.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.444+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f357409be10 con 0x7f3584102980 2024-04-22T01:11:00.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.658+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f3568000bf0 con 0x7f35700a2c10 2024-04-22T01:11:00.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.664+0000 7f357b7fe640 1 -- 172.21.15.38:0/1419710701 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3568000bf0 con 0x7f35700a2c10 2024-04-22T01:11:00.664 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/qa/sv_0/15047a62-d907-4950-9c83-90e9c7e0dc96 2024-04-22T01:11:00.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.665+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35700a2c10 msgr2=0x7f35700a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.665+0000 7f358c3a0640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35700a2c10 0x7f35700a50d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f358002c8e0 tx=0x7f358002c670 comp rx=0 tx=0).stop 2024-04-22T01:11:00.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.665+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 msgr2=0x7f3584198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.665+0000 7f358c3a0640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 0x7f3584198040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f357400e9f0 tx=0x7f357400eeb0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 shutdown_connections 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35700a2c10 0x7f35700a50d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3584103340 0x7f358419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3584102980 0x7f3584198040 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 --2- 172.21.15.38:0/1419710701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3584101790 0x7f3584197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 >> 172.21.15.38:0/1419710701 conn(0x7f35840fcd20 msgr2=0x7f3584105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 shutdown_connections 2024-04-22T01:11:00.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.666+0000 7f358c3a0640 1 -- 172.21.15.38:0/1419710701 wait complete. 2024-04-22T01:11:00.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/15768481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 msgr2=0x7f3ae0109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/15768481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae0109be0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009ca0 tx=0x7f3ad802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/15768481 shutdown_connections 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/15768481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae0109be0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/15768481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae0102990 0x7f3ae0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/15768481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae01017a0 0x7f3ae0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/15768481 >> 172.21.15.38:0/15768481 conn(0x7f3ae00fcd10 msgr2=0x7f3ae00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/15768481 shutdown_connections 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.783+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/15768481 wait complete. 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 Processor -- start 2024-04-22T01:11:00.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 -- start start 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae01017a0 0x7f3ae0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae0102990 0x7f3ae0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae010d400 con 0x7f3ae0103350 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae010d280 con 0x7f3ae0102990 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3ae5b8a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae010d580 con 0x7f3ae01017a0 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3adffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3adffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44428/0 (socket says 172.21.15.38:44428) 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.784+0000 7f3adffff640 1 -- 172.21.15.38:0/570481498 learned_addr learned my addr 172.21.15.38:0/570481498 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adeffd640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae0102990 0x7f3ae0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adf7fe640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae01017a0 0x7f3ae0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adffff640 1 -- 172.21.15.38:0/570481498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae01017a0 msgr2=0x7f3ae0197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adffff640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae01017a0 0x7f3ae0197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adffff640 1 -- 172.21.15.38:0/570481498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae0102990 msgr2=0x7f3ae0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adffff640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae0102990 0x7f3ae0197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adffff640 1 -- 172.21.15.38:0/570481498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae019f660 con 0x7f3ae0103350 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adffff640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae019eff0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8037040 tx=0x7f3ad8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8041020 con 0x7f3ae0103350 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae019f8f0 con 0x7f3ae0103350 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ad8034d90 con 0x7f3ae0103350 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8039040 con 0x7f3ae0103350 2024-04-22T01:11:00.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.785+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae019fde0 con 0x7f3ae0103350 2024-04-22T01:11:00.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.786+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae0108ac0 con 0x7f3ae0103350 2024-04-22T01:11:00.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.787+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3ad8044020 con 0x7f3ae0103350 2024-04-22T01:11:00.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.788+0000 7f3adcff9640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ac80a2b90 0x7f3ac80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:00.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.788+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3ad80f1230 con 0x7f3ae0103350 2024-04-22T01:11:00.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.788+0000 7f3adf7fe640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ac80a2b90 0x7f3ac80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:00.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.788+0000 7f3adf7fe640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ac80a2b90 0x7f3ac80a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0005db0 tx=0x7f3ad0005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:00.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:00.792+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3ad804b020 con 0x7f3ae0103350 2024-04-22T01:11:01.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.007+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3ae0061c10 con 0x7f3ac80a2b90 2024-04-22T01:11:01.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.032+0000 7f3adcff9640 1 -- 172.21.15.38:0/570481498 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3ae0061c10 con 0x7f3ac80a2b90 2024-04-22T01:11:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ac80a2b90 msgr2=0x7f3ac80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ac80a2b90 0x7f3ac80a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0005db0 tx=0x7f3ad0005d40 comp rx=0 tx=0).stop 2024-04-22T01:11:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 msgr2=0x7f3ae019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae019eff0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8037040 tx=0x7f3ad8032f20 comp rx=0 tx=0).stop 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 shutdown_connections 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ac80a2b90 0x7f3ac80a5050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ae0103350 0x7f3ae019eff0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae0102990 0x7f3ae0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 --2- 172.21.15.38:0/570481498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae01017a0 0x7f3ae0197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 >> 172.21.15.38:0/570481498 conn(0x7f3ae00fcd10 msgr2=0x7f3ae0105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 shutdown_connections 2024-04-22T01:11:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.035+0000 7f3ae5b8a640 1 -- 172.21.15.38:0/570481498 wait complete. 2024-04-22T01:11:01.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: pgmap v311: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-22T01:11:01.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] up:active 2024-04-22T01:11:01.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T01:11:01.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: Cluster is now healthy 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] up:active 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T01:11:01.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:00 smithi038 ceph-mon[28043]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:01.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 -- 172.21.15.38:0/2641756867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c101870 msgr2=0x7f444c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 --2- 172.21.15.38:0/2641756867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c101870 0x7f444c101cf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f443c009900 tx=0x7f443c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 -- 172.21.15.38:0/2641756867 shutdown_connections 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 --2- 172.21.15.38:0/2641756867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f444c102230 0x7f444c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 --2- 172.21.15.38:0/2641756867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c101870 0x7f444c101cf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 --2- 172.21.15.38:0/2641756867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f444c100680 0x7f444c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 -- 172.21.15.38:0/2641756867 >> 172.21.15.38:0/2641756867 conn(0x7f444c0fbbf0 msgr2=0x7f444c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 -- 172.21.15.38:0/2641756867 shutdown_connections 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.153+0000 7f4454435640 1 -- 172.21.15.38:0/2641756867 wait complete. 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 Processor -- start 2024-04-22T01:11:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 -- start start 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 0x7f444c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f444c101870 0x7f444c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f444c102230 0x7f444c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f444c10c2e0 con 0x7f444c100680 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f444c10c160 con 0x7f444c101870 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f4454435640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f444c10c460 con 0x7f444c102230 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f44521d1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 0x7f444c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f44521d1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 0x7f444c104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44446/0 (socket says 172.21.15.38:44446) 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f44521d1640 1 -- 172.21.15.38:0/727606111 learned_addr learned my addr 172.21.15.38:0/727606111 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f44519d0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f444c101870 0x7f444c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.154+0000 7f44529d2640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f444c102230 0x7f444c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44521d1640 1 -- 172.21.15.38:0/727606111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f444c102230 msgr2=0x7f444c103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44521d1640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f444c102230 0x7f444c103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44521d1640 1 -- 172.21.15.38:0/727606111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f444c101870 msgr2=0x7f444c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44521d1640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f444c101870 0x7f444c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44521d1640 1 -- 172.21.15.38:0/727606111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c103a20 con 0x7f444c100680 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44521d1640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 0x7f444c104610 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f444800d770 tx=0x7f444800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448017070 con 0x7f444c100680 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444c1a1c30 con 0x7f444c100680 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4448009c30 con 0x7f444c100680 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448003dd0 con 0x7f444c100680 2024-04-22T01:11:01.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.155+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444c1a21a0 con 0x7f444c100680 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.156+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444c1a2370 con 0x7f444c100680 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.157+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4448003f70 con 0x7f444c100680 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.158+0000 7f44437fe640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44380a2b90 0x7f44380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.158+0000 7f44519d0640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44380a2b90 0x7f44380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.158+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f44480cebf0 con 0x7f444c100680 2024-04-22T01:11:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.158+0000 7f44519d0640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44380a2b90 0x7f44380a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f443c02c7b0 tx=0x7f443c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:01.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.163+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4448009de0 con 0x7f444c100680 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: pgmap v311: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] up:active 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: Cluster is now healthy 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] up:active 2024-04-22T01:11:01.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T01:11:01.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:00 smithi203 ceph-mon[31825]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:01.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.378+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f444c0fde40 con 0x7f44380a2b90 2024-04-22T01:11:01.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.382+0000 7f44437fe640 1 -- 172.21.15.38:0/727606111 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f444c0fde40 con 0x7f44380a2b90 2024-04-22T01:11:01.383 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/qa/sv_1/2169144e-b167-4a60-86e4-9fba173df7da 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.384+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44380a2b90 msgr2=0x7f44380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.384+0000 7f4454435640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44380a2b90 0x7f44380a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f443c02c7b0 tx=0x7f443c005fd0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.384+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 msgr2=0x7f444c104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.384+0000 7f4454435640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 0x7f444c104610 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f444800d770 tx=0x7f444800dc30 comp rx=0 tx=0).stop 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 shutdown_connections 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f44380a2b90 0x7f44380a5050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f444c102230 0x7f444c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f444c101870 0x7f444c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 --2- 172.21.15.38:0/727606111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f444c100680 0x7f444c104610 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 >> 172.21.15.38:0/727606111 conn(0x7f444c0fbbf0 msgr2=0x7f444c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 shutdown_connections 2024-04-22T01:11:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.385+0000 7f4454435640 1 -- 172.21.15.38:0/727606111 wait complete. 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: pgmap v311: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] up:active 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: Cluster is now healthy 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-04-22T01:11:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-22T01:11:01.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-04-22T01:11:01.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:01.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] up:active 2024-04-22T01:11:01.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T01:11:01.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:00 smithi111 ceph-mon[31057]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:01.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 -- 172.21.15.38:0/3886187381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4102250 msgr2=0x7f2ce4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3886187381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4102250 0x7f2ce4108ae0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009cc0 tx=0x7f2cdc030510 comp rx=0 tx=0).stop 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 -- 172.21.15.38:0/3886187381 shutdown_connections 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3886187381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4102250 0x7f2ce4108ae0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3886187381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce4101890 0x7f2ce4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3886187381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce41006a0 0x7f2ce4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 -- 172.21.15.38:0/3886187381 >> 172.21.15.38:0/3886187381 conn(0x7f2ce40fbc30 msgr2=0x7f2ce40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 -- 172.21.15.38:0/3886187381 shutdown_connections 2024-04-22T01:11:01.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.499+0000 7f2cea44b640 1 -- 172.21.15.38:0/3886187381 wait complete. 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 Processor -- start 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 -- start start 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce41006a0 0x7f2ce4195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 0x7f2ce4195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce4102250 0x7f2ce419ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce410b230 con 0x7f2ce4101890 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce410b0b0 con 0x7f2ce4102250 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2cea44b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce410b3b0 con 0x7f2ce41006a0 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2ce37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 0x7f2ce4195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2ce37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 0x7f2ce4195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44466/0 (socket says 172.21.15.38:44466) 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.500+0000 7f2ce37fe640 1 -- 172.21.15.38:0/3906527887 learned_addr learned my addr 172.21.15.38:0/3906527887 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce3fff640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce41006a0 0x7f2ce4195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce89e8640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce4102250 0x7f2ce419ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce37fe640 1 -- 172.21.15.38:0/3906527887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce41006a0 msgr2=0x7f2ce4195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce37fe640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce41006a0 0x7f2ce4195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce37fe640 1 -- 172.21.15.38:0/3906527887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce4102250 msgr2=0x7f2ce419ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce37fe640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce4102250 0x7f2ce419ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce37fe640 1 -- 172.21.15.38:0/3906527887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce419d4b0 con 0x7f2ce4101890 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce89e8640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce4102250 0x7f2ce419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce3fff640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce41006a0 0x7f2ce4195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce37fe640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 0x7f2ce4195dd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800a8d0 tx=0x7f2cd800ad90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80079b0 con 0x7f2ce4101890 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce419d7a0 con 0x7f2ce4101890 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd8005500 con 0x7f2ce4101890 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8014410 con 0x7f2ce4101890 2024-04-22T01:11:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.501+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce419dcf0 con 0x7f2ce4101890 2024-04-22T01:11:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.502+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce4107960 con 0x7f2ce4101890 2024-04-22T01:11:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.503+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2cd8004e70 con 0x7f2ce4101890 2024-04-22T01:11:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.504+0000 7f2ce17fa640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2b90 0x7f2ccc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.504+0000 7f2ce3fff640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2b90 0x7f2ccc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.504+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2cd80ce810 con 0x7f2ce4101890 2024-04-22T01:11:01.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.504+0000 7f2ce3fff640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2b90 0x7f2ccc0a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40058c0 tx=0x7f2cd4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.508+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2cd809b4d0 con 0x7f2ce4101890 2024-04-22T01:11:01.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.732+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ce4062340 con 0x7f2ce4101890 2024-04-22T01:11:01.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.732+0000 7f2ce17fa640 1 -- 172.21.15.38:0/3906527887 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f2cd809add0 con 0x7f2ce4101890 2024-04-22T01:11:01.733 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:01.733 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:11:00.429607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24451,24472,24446,24527,24485]},"id":1}]} 2024-04-22T01:11:01.733 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.734+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2b90 msgr2=0x7f2ccc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.734+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2b90 0x7f2ccc0a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40058c0 tx=0x7f2cd4005850 comp rx=0 tx=0).stop 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.734+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 msgr2=0x7f2ce4195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.734+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 0x7f2ce4195dd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800a8d0 tx=0x7f2cd800ad90 comp rx=0 tx=0).stop 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 shutdown_connections 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2b90 0x7f2ccc0a5050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce4102250 0x7f2ce419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce4101890 0x7f2ce4195dd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 --2- 172.21.15.38:0/3906527887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce41006a0 0x7f2ce4195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 >> 172.21.15.38:0/3906527887 conn(0x7f2ce40fbc30 msgr2=0x7f2ce4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:01.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 shutdown_connections 2024-04-22T01:11:01.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.735+0000 7f2cea44b640 1 -- 172.21.15.38:0/3906527887 wait complete. 2024-04-22T01:11:01.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:11:01.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 -- 172.21.15.38:0/2390429946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c101870 msgr2=0x7f839c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 --2- 172.21.15.38:0/2390429946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c101870 0x7f839c101cf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f838c009900 tx=0x7f838c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:11:01.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 -- 172.21.15.38:0/2390429946 shutdown_connections 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 --2- 172.21.15.38:0/2390429946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c102230 0x7f839c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 --2- 172.21.15.38:0/2390429946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c101870 0x7f839c101cf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 --2- 172.21.15.38:0/2390429946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c100680 0x7f839c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 -- 172.21.15.38:0/2390429946 >> 172.21.15.38:0/2390429946 conn(0x7f839c0fbbf0 msgr2=0x7f839c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 -- 172.21.15.38:0/2390429946 shutdown_connections 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 -- 172.21.15.38:0/2390429946 wait complete. 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 Processor -- start 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.848+0000 7f83a4496640 1 -- start start 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c100680 0x7f839c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 0x7f839c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10c2e0 con 0x7f839c102230 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10c160 con 0x7f839c100680 2024-04-22T01:11:01.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10c460 con 0x7f839c101870 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 0x7f839c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2232640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c100680 0x7f839c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 0x7f839c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44490/0 (socket says 172.21.15.38:44490) 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 -- 172.21.15.38:0/44614490 learned_addr learned my addr 172.21.15.38:0/44614490 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a1a31640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 -- 172.21.15.38:0/44614490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 msgr2=0x7f839c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 -- 172.21.15.38:0/44614490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c100680 msgr2=0x7f839c104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c100680 0x7f839c104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 -- 172.21.15.38:0/44614490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c103a20 con 0x7f839c102230 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a1a31640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a2a33640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 0x7f839c103190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f839400cad0 tx=0x7f839400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839400ae20 con 0x7f839c102230 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8394005020 con 0x7f839c102230 2024-04-22T01:11:01.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c1a1bd0 con 0x7f839c102230 2024-04-22T01:11:01.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.849+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839400f6f0 con 0x7f839c102230 2024-04-22T01:11:01.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.850+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c1a20f0 con 0x7f839c102230 2024-04-22T01:11:01.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.851+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839c068560 con 0x7f839c102230 2024-04-22T01:11:01.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.852+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f83940051d0 con 0x7f839c102230 2024-04-22T01:11:01.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.852+0000 7f83937fe640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83880a2c10 0x7f83880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:01.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.852+0000 7f83a2232640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83880a2c10 0x7f83880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:01.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.852+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f83940cf230 con 0x7f839c102230 2024-04-22T01:11:01.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.853+0000 7f83a2232640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83880a2c10 0x7f83880a50d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f839c1016d0 tx=0x7f839800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:01.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.857+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8394014030 con 0x7f839c102230 2024-04-22T01:11:01.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:01.940+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8394097f60 con 0x7f839c102230 2024-04-22T01:11:02.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.049+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f839c1a25c0 con 0x7f839c102230 2024-04-22T01:11:02.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.050+0000 7f83937fe640 1 -- 172.21.15.38:0/44614490 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f83940c72a0 con 0x7f839c102230 2024-04-22T01:11:02.051 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:02.051 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.052+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83880a2c10 msgr2=0x7f83880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.052+0000 7f83a4496640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83880a2c10 0x7f83880a50d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f839c1016d0 tx=0x7f839800a2d0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.052+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 msgr2=0x7f839c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.052+0000 7f83a4496640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 0x7f839c103190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f839400cad0 tx=0x7f839400cf90 comp rx=0 tx=0).stop 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 shutdown_connections 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83880a2c10 0x7f83880a50d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c102230 0x7f839c103190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 --2- 172.21.15.38:0/44614490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c100680 0x7f839c104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 >> 172.21.15.38:0/44614490 conn(0x7f839c0fbbf0 msgr2=0x7f839c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:02.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 shutdown_connections 2024-04-22T01:11:02.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.053+0000 7f83a4496640 1 -- 172.21.15.38:0/44614490 wait complete. 2024-04-22T01:11:02.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:01 smithi038 ceph-mon[28043]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:02.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:01 smithi038 ceph-mon[28043]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:02.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:01 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3906527887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:11:02.064 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T01:11:02.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:11:02.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 -- 172.21.15.38:0/413087911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 msgr2=0x7efdb0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 --2- 172.21.15.38:0/413087911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0108ac0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efda8009ca0 tx=0x7efda802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 -- 172.21.15.38:0/413087911 shutdown_connections 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 --2- 172.21.15.38:0/413087911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0108ac0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 --2- 172.21.15.38:0/413087911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdb0101870 0x7efdb0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 --2- 172.21.15.38:0/413087911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 0x7efdb0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 -- 172.21.15.38:0/413087911 >> 172.21.15.38:0/413087911 conn(0x7efdb00fbbf0 msgr2=0x7efdb00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 -- 172.21.15.38:0/413087911 shutdown_connections 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.167+0000 7efdb6165640 1 -- 172.21.15.38:0/413087911 wait complete. 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 Processor -- start 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 -- start start 2024-04-22T01:11:02.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 0x7efdb0104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdb0101870 0x7efdb0102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010c2e0 con 0x7efdb0102230 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010c160 con 0x7efdb0100680 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdb6165640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010c460 con 0x7efdb0101870 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdaffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdaffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44498/0 (socket says 172.21.15.38:44498) 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdaffff640 1 -- 172.21.15.38:0/485030545 learned_addr learned my addr 172.21.15.38:0/485030545 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.168+0000 7efdaeffd640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdb0101870 0x7efdb0102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaf7fe640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 0x7efdb0104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaffff640 1 -- 172.21.15.38:0/485030545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdb0101870 msgr2=0x7efdb0102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaffff640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdb0101870 0x7efdb0102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaffff640 1 -- 172.21.15.38:0/485030545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 msgr2=0x7efdb0104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaffff640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 0x7efdb0104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaffff640 1 -- 172.21.15.38:0/485030545 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb0103a20 con 0x7efdb0102230 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaf7fe640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 0x7efdb0104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdaffff640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0103190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efda8037040 tx=0x7efda8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda8041020 con 0x7efdb0102230 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda8034d90 con 0x7efdb0102230 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb01a1b00 con 0x7efdb0102230 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda8039040 con 0x7efdb0102230 2024-04-22T01:11:02.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.169+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb01a2050 con 0x7efdb0102230 2024-04-22T01:11:02.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.170+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb01a2220 con 0x7efdb0102230 2024-04-22T01:11:02.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.171+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7efda8044020 con 0x7efdb0102230 2024-04-22T01:11:02.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.172+0000 7efdacff9640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd980a2c20 0x7efd980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.172+0000 7efdaf7fe640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd980a2c20 0x7efd980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.172+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efda80f1360 con 0x7efdb0102230 2024-04-22T01:11:02.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.172+0000 7efdaf7fe640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd980a2c20 0x7efd980a50e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efda0005d00 tx=0x7efda0005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:02.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.176+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efda804b020 con 0x7efdb0102230 2024-04-22T01:11:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:01 smithi203 ceph-mon[31825]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:01 smithi203 ceph-mon[31825]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:02.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:01 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3906527887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:11:02.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.358+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efdb01a23f0 con 0x7efd980a2c20 2024-04-22T01:11:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.369+0000 7efdacff9640 1 -- 172.21.15.38:0/485030545 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340633 (secure 0 0 0) 0x7efdb01a23f0 con 0x7efd980a2c20 2024-04-22T01:11:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:02.376 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:11:02.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd980a2c20 msgr2=0x7efd980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd980a2c20 0x7efd980a50e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efda0005d00 tx=0x7efda0005ef0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 msgr2=0x7efdb0103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0103190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efda8037040 tx=0x7efda8032f20 comp rx=0 tx=0).stop 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 shutdown_connections 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efd980a2c20 0x7efd980a50e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdb0102230 0x7efdb0103190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdb0101870 0x7efdb0102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 --2- 172.21.15.38:0/485030545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb0100680 0x7efdb0104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.372+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 >> 172.21.15.38:0/485030545 conn(0x7efdb00fbbf0 msgr2=0x7efdb00fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.373+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 shutdown_connections 2024-04-22T01:11:02.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.373+0000 7efdb6165640 1 -- 172.21.15.38:0/485030545 wait complete. 2024-04-22T01:11:02.384 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-04-22T01:11:01.790535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":641597,"num_objects":78,"num_object_clones":0,"num_object_copies":235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":149,"num_read_kb":172,"num_write":358,"num_write_kb":3043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":292,"ondisk_log_size":292,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356736,"kb_used_data":11736,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585088,"statfs":{"total":1151940427776,"available":1151575130112,"internally_reserved":0,"allocated":12017664,"data_stored":7567681,"data_compressed":285387,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46730,"num_objects":54,"num_object_clones":0,"num_object_copies":163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":23,"num_read_kb":63,"num_write":130,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003909"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176576+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:10:36.176576+0000","last_peered":"2024-04-22T01:10:36.176576+0000","last_clean":"2024-04-22T01:10:36.176576+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:10:36.176576+0000","last_undegraded":"2024-04-22T01:10:36.176576+0000","last_fullsized":"2024-04-22T01:10:36.176576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175529+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:10:36.175529+0000","last_peered":"2024-04-22T01:10:36.175529+0000","last_clean":"2024-04-22T01:10:36.175529+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:10:36.175529+0000","last_undegraded":"2024-04-22T01:10:36.175529+0000","last_fullsized":"2024-04-22T01:10:36.175529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175264+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:10:36.175264+0000","last_peered":"2024-04-22T01:10:36.175264+0000","last_clean":"2024-04-22T01:10:36.175264+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:10:36.175264+0000","last_undegraded":"2024-04-22T01:10:36.175264+0000","last_fullsized":"2024-04-22T01:10:36.175264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967267+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:10:35.967267+0000","last_peered":"2024-04-22T01:10:35.967267+0000","last_clean":"2024-04-22T01:10:35.967267+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:10:35.967267+0000","last_undegraded":"2024-04-22T01:10:35.967267+0000","last_fullsized":"2024-04-22T01:10:35.967267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175960+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:10:36.175960+0000","last_peered":"2024-04-22T01:10:36.175960+0000","last_clean":"2024-04-22T01:10:36.175960+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:10:36.175960+0000","last_undegraded":"2024-04-22T01:10:36.175960+0000","last_fullsized":"2024-04-22T01:10:36.175960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175929+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:10:36.175929+0000","last_peered":"2024-04-22T01:10:36.175929+0000","last_clean":"2024-04-22T01:10:36.175929+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:10:36.175929+0000","last_undegraded":"2024-04-22T01:10:36.175929+0000","last_fullsized":"2024-04-22T01:10:36.175929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429231+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:10:35.429231+0000","last_peered":"2024-04-22T01:10:35.429231+0000","last_clean":"2024-04-22T01:10:35.429231+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:10:35.429231+0000","last_undegraded":"2024-04-22T01:10:35.429231+0000","last_fullsized":"2024-04-22T01:10:35.429231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175589+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:10:36.175589+0000","last_peered":"2024-04-22T01:10:36.175589+0000","last_clean":"2024-04-22T01:10:36.175589+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:10:36.175589+0000","last_undegraded":"2024-04-22T01:10:36.175589+0000","last_fullsized":"2024-04-22T01:10:36.175589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176623+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:10:36.176623+0000","last_peered":"2024-04-22T01:10:36.176623+0000","last_clean":"2024-04-22T01:10:36.176623+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:10:36.176623+0000","last_undegraded":"2024-04-22T01:10:36.176623+0000","last_fullsized":"2024-04-22T01:10:36.176623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176540+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:10:36.176540+0000","last_peered":"2024-04-22T01:10:36.176540+0000","last_clean":"2024-04-22T01:10:36.176540+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:10:36.176540+0000","last_undegraded":"2024-04-22T01:10:36.176540+0000","last_fullsized":"2024-04-22T01:10:36.176540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429879+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:10:35.429879+0000","last_peered":"2024-04-22T01:10:35.429879+0000","last_clean":"2024-04-22T01:10:35.429879+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:10:35.429879+0000","last_undegraded":"2024-04-22T01:10:35.429879+0000","last_fullsized":"2024-04-22T01:10:35.429879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967105+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:10:35.967105+0000","last_peered":"2024-04-22T01:10:35.967105+0000","last_clean":"2024-04-22T01:10:35.967105+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:10:35.967105+0000","last_undegraded":"2024-04-22T01:10:35.967105+0000","last_fullsized":"2024-04-22T01:10:35.967105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772199+0000","last_change":"2024-04-22T01:10:30.452385+0000","last_active":"2024-04-22T01:10:35.772199+0000","last_peered":"2024-04-22T01:10:35.772199+0000","last_clean":"2024-04-22T01:10:35.772199+0000","last_became_active":"2024-04-22T01:10:30.452034+0000","last_became_peered":"2024-04-22T01:10:30.452034+0000","last_unstale":"2024-04-22T01:10:35.772199+0000","last_undegraded":"2024-04-22T01:10:35.772199+0000","last_fullsized":"2024-04-22T01:10:35.772199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:14:02.046478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.174949+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:10:36.174949+0000","last_peered":"2024-04-22T01:10:36.174949+0000","last_clean":"2024-04-22T01:10:36.174949+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:10:36.174949+0000","last_undegraded":"2024-04-22T01:10:36.174949+0000","last_fullsized":"2024-04-22T01:10:36.174949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.174991+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:10:36.174991+0000","last_peered":"2024-04-22T01:10:36.174991+0000","last_clean":"2024-04-22T01:10:36.174991+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:10:36.174991+0000","last_undegraded":"2024-04-22T01:10:36.174991+0000","last_fullsized":"2024-04-22T01:10:36.174991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175718+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:10:36.175718+0000","last_peered":"2024-04-22T01:10:36.175718+0000","last_clean":"2024-04-22T01:10:36.175718+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:10:36.175718+0000","last_undegraded":"2024-04-22T01:10:36.175718+0000","last_fullsized":"2024-04-22T01:10:36.175718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176239+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:10:36.176239+0000","last_peered":"2024-04-22T01:10:36.176239+0000","last_clean":"2024-04-22T01:10:36.176239+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:10:36.176239+0000","last_undegraded":"2024-04-22T01:10:36.176239+0000","last_fullsized":"2024-04-22T01:10:36.176239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431380+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:10:35.431380+0000","last_peered":"2024-04-22T01:10:35.431380+0000","last_clean":"2024-04-22T01:10:35.431380+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:10:35.431380+0000","last_undegraded":"2024-04-22T01:10:35.431380+0000","last_fullsized":"2024-04-22T01:10:35.431380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429794+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:10:35.429794+0000","last_peered":"2024-04-22T01:10:35.429794+0000","last_clean":"2024-04-22T01:10:35.429794+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:10:35.429794+0000","last_undegraded":"2024-04-22T01:10:35.429794+0000","last_fullsized":"2024-04-22T01:10:35.429794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429385+0000","last_change":"2024-04-22T01:10:34.667483+0000","last_active":"2024-04-22T01:10:35.429385+0000","last_peered":"2024-04-22T01:10:35.429385+0000","last_clean":"2024-04-22T01:10:35.429385+0000","last_became_active":"2024-04-22T01:10:34.666997+0000","last_became_peered":"2024-04-22T01:10:34.666997+0000","last_unstale":"2024-04-22T01:10:35.429385+0000","last_undegraded":"2024-04-22T01:10:35.429385+0000","last_fullsized":"2024-04-22T01:10:35.429385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:03:35.317553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:00.441670+0000","last_change":"2024-04-22T01:10:29.585069+0000","last_active":"2024-04-22T01:11:00.441670+0000","last_peered":"2024-04-22T01:11:00.441670+0000","last_clean":"2024-04-22T01:11:00.441670+0000","last_became_active":"2024-04-22T01:10:29.584651+0000","last_became_peered":"2024-04-22T01:10:29.584651+0000","last_unstale":"2024-04-22T01:11:00.441670+0000","last_undegraded":"2024-04-22T01:11:00.441670+0000","last_fullsized":"2024-04-22T01:11:00.441670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:13.121214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176127+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:10:36.176127+0000","last_peered":"2024-04-22T01:10:36.176127+0000","last_clean":"2024-04-22T01:10:36.176127+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:10:36.176127+0000","last_undegraded":"2024-04-22T01:10:36.176127+0000","last_fullsized":"2024-04-22T01:10:36.176127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.966796+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:10:35.966796+0000","last_peered":"2024-04-22T01:10:35.966796+0000","last_clean":"2024-04-22T01:10:35.966796+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:10:35.966796+0000","last_undegraded":"2024-04-22T01:10:35.966796+0000","last_fullsized":"2024-04-22T01:10:35.966796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176561+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:10:36.176561+0000","last_peered":"2024-04-22T01:10:36.176561+0000","last_clean":"2024-04-22T01:10:36.176561+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:10:36.176561+0000","last_undegraded":"2024-04-22T01:10:36.176561+0000","last_fullsized":"2024-04-22T01:10:36.176561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175408+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:10:36.175408+0000","last_peered":"2024-04-22T01:10:36.175408+0000","last_clean":"2024-04-22T01:10:36.175408+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:10:36.175408+0000","last_undegraded":"2024-04-22T01:10:36.175408+0000","last_fullsized":"2024-04-22T01:10:36.175408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.966823+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:10:35.966823+0000","last_peered":"2024-04-22T01:10:35.966823+0000","last_clean":"2024-04-22T01:10:35.966823+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:10:35.966823+0000","last_undegraded":"2024-04-22T01:10:35.966823+0000","last_fullsized":"2024-04-22T01:10:35.966823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:57.905051+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:10:57.905051+0000","last_peered":"2024-04-22T01:10:57.905051+0000","last_clean":"2024-04-22T01:10:57.905051+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:10:57.905051+0000","last_undegraded":"2024-04-22T01:10:57.905051+0000","last_fullsized":"2024-04-22T01:10:57.905051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176403+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:10:36.176403+0000","last_peered":"2024-04-22T01:10:36.176403+0000","last_clean":"2024-04-22T01:10:36.176403+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:10:36.176403+0000","last_undegraded":"2024-04-22T01:10:36.176403+0000","last_fullsized":"2024-04-22T01:10:36.176403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.966821+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:10:35.966821+0000","last_peered":"2024-04-22T01:10:35.966821+0000","last_clean":"2024-04-22T01:10:35.966821+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:10:35.966821+0000","last_undegraded":"2024-04-22T01:10:35.966821+0000","last_fullsized":"2024-04-22T01:10:35.966821+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176322+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:10:36.176322+0000","last_peered":"2024-04-22T01:10:36.176322+0000","last_clean":"2024-04-22T01:10:36.176322+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:10:36.176322+0000","last_undegraded":"2024-04-22T01:10:36.176322+0000","last_fullsized":"2024-04-22T01:10:36.176322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507530+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:10:35.507530+0000","last_peered":"2024-04-22T01:10:35.507530+0000","last_clean":"2024-04-22T01:10:35.507530+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:10:35.507530+0000","last_undegraded":"2024-04-22T01:10:35.507530+0000","last_fullsized":"2024-04-22T01:10:35.507530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429495+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:10:35.429495+0000","last_peered":"2024-04-22T01:10:35.429495+0000","last_clean":"2024-04-22T01:10:35.429495+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:10:35.429495+0000","last_undegraded":"2024-04-22T01:10:35.429495+0000","last_fullsized":"2024-04-22T01:10:35.429495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.893426+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:10:56.893426+0000","last_peered":"2024-04-22T01:10:56.893426+0000","last_clean":"2024-04-22T01:10:56.893426+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:10:56.893426+0000","last_undegraded":"2024-04-22T01:10:56.893426+0000","last_fullsized":"2024-04-22T01:10:56.893426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175921+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:10:36.175921+0000","last_peered":"2024-04-22T01:10:36.175921+0000","last_clean":"2024-04-22T01:10:36.175921+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:10:36.175921+0000","last_undegraded":"2024-04-22T01:10:36.175921+0000","last_fullsized":"2024-04-22T01:10:36.175921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429594+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:10:35.429594+0000","last_peered":"2024-04-22T01:10:35.429594+0000","last_clean":"2024-04-22T01:10:35.429594+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:10:35.429594+0000","last_undegraded":"2024-04-22T01:10:35.429594+0000","last_fullsized":"2024-04-22T01:10:35.429594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.914913+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:10:58.914913+0000","last_peered":"2024-04-22T01:10:58.914913+0000","last_clean":"2024-04-22T01:10:58.914913+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:10:58.914913+0000","last_undegraded":"2024-04-22T01:10:58.914913+0000","last_fullsized":"2024-04-22T01:10:58.914913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507706+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:10:35.507706+0000","last_peered":"2024-04-22T01:10:35.507706+0000","last_clean":"2024-04-22T01:10:35.507706+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:10:35.507706+0000","last_undegraded":"2024-04-22T01:10:35.507706+0000","last_fullsized":"2024-04-22T01:10:35.507706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176059+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:10:36.176059+0000","last_peered":"2024-04-22T01:10:36.176059+0000","last_clean":"2024-04-22T01:10:36.176059+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:10:36.176059+0000","last_undegraded":"2024-04-22T01:10:36.176059+0000","last_fullsized":"2024-04-22T01:10:36.176059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772110+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:10:35.772110+0000","last_peered":"2024-04-22T01:10:35.772110+0000","last_clean":"2024-04-22T01:10:35.772110+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:10:35.772110+0000","last_undegraded":"2024-04-22T01:10:35.772110+0000","last_fullsized":"2024-04-22T01:10:35.772110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175949+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:10:36.175949+0000","last_peered":"2024-04-22T01:10:36.175949+0000","last_clean":"2024-04-22T01:10:36.175949+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:10:36.175949+0000","last_undegraded":"2024-04-22T01:10:36.175949+0000","last_fullsized":"2024-04-22T01:10:36.175949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175475+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:10:36.175475+0000","last_peered":"2024-04-22T01:10:36.175475+0000","last_clean":"2024-04-22T01:10:36.175475+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:10:36.175475+0000","last_undegraded":"2024-04-22T01:10:36.175475+0000","last_fullsized":"2024-04-22T01:10:36.175475+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891531+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:10:56.891531+0000","last_peered":"2024-04-22T01:10:56.891531+0000","last_clean":"2024-04-22T01:10:56.891531+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:10:56.891531+0000","last_undegraded":"2024-04-22T01:10:56.891531+0000","last_fullsized":"2024-04-22T01:10:56.891531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.430603+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:10:35.430603+0000","last_peered":"2024-04-22T01:10:35.430603+0000","last_clean":"2024-04-22T01:10:35.430603+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:10:35.430603+0000","last_undegraded":"2024-04-22T01:10:35.430603+0000","last_fullsized":"2024-04-22T01:10:35.430603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176153+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:10:36.176153+0000","last_peered":"2024-04-22T01:10:36.176153+0000","last_clean":"2024-04-22T01:10:36.176153+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:10:36.176153+0000","last_undegraded":"2024-04-22T01:10:36.176153+0000","last_fullsized":"2024-04-22T01:10:36.176153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:01.440127+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:11:01.440127+0000","last_peered":"2024-04-22T01:11:01.440127+0000","last_clean":"2024-04-22T01:11:01.440127+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:11:01.440127+0000","last_undegraded":"2024-04-22T01:11:01.440127+0000","last_fullsized":"2024-04-22T01:11:01.440127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176183+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:10:36.176183+0000","last_peered":"2024-04-22T01:10:36.176183+0000","last_clean":"2024-04-22T01:10:36.176183+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:10:36.176183+0000","last_undegraded":"2024-04-22T01:10:36.176183+0000","last_fullsized":"2024-04-22T01:10:36.176183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.430820+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:10:35.430820+0000","last_peered":"2024-04-22T01:10:35.430820+0000","last_clean":"2024-04-22T01:10:35.430820+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:10:35.430820+0000","last_undegraded":"2024-04-22T01:10:35.430820+0000","last_fullsized":"2024-04-22T01:10:35.430820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891351+0000","last_change":"2024-04-22T01:10:29.591793+0000","last_active":"2024-04-22T01:10:56.891351+0000","last_peered":"2024-04-22T01:10:56.891351+0000","last_clean":"2024-04-22T01:10:56.891351+0000","last_became_active":"2024-04-22T01:10:29.591245+0000","last_became_peered":"2024-04-22T01:10:29.591245+0000","last_unstale":"2024-04-22T01:10:56.891351+0000","last_undegraded":"2024-04-22T01:10:56.891351+0000","last_fullsized":"2024-04-22T01:10:56.891351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:40:28.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175366+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:10:36.175366+0000","last_peered":"2024-04-22T01:10:36.175366+0000","last_clean":"2024-04-22T01:10:36.175366+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:10:36.175366+0000","last_undegraded":"2024-04-22T01:10:36.175366+0000","last_fullsized":"2024-04-22T01:10:36.175366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967313+0000","last_change":"2024-04-22T01:10:34.671540+0000","last_active":"2024-04-22T01:10:35.967313+0000","last_peered":"2024-04-22T01:10:35.967313+0000","last_clean":"2024-04-22T01:10:35.967313+0000","last_became_active":"2024-04-22T01:10:34.670777+0000","last_became_peered":"2024-04-22T01:10:34.670777+0000","last_unstale":"2024-04-22T01:10:35.967313+0000","last_undegraded":"2024-04-22T01:10:35.967313+0000","last_fullsized":"2024-04-22T01:10:35.967313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:32.654313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.890841+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:10:56.890841+0000","last_peered":"2024-04-22T01:10:56.890841+0000","last_clean":"2024-04-22T01:10:56.890841+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:10:56.890841+0000","last_undegraded":"2024-04-22T01:10:56.890841+0000","last_fullsized":"2024-04-22T01:10:56.890841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176492+0000","last_change":"2024-04-22T01:10:30.473144+0000","last_active":"2024-04-22T01:10:36.176492+0000","last_peered":"2024-04-22T01:10:36.176492+0000","last_clean":"2024-04-22T01:10:36.176492+0000","last_became_active":"2024-04-22T01:10:30.472860+0000","last_became_peered":"2024-04-22T01:10:30.472860+0000","last_unstale":"2024-04-22T01:10:36.176492+0000","last_undegraded":"2024-04-22T01:10:36.176492+0000","last_fullsized":"2024-04-22T01:10:36.176492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:14.097318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.430803+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:10:35.430803+0000","last_peered":"2024-04-22T01:10:35.430803+0000","last_clean":"2024-04-22T01:10:35.430803+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:10:35.430803+0000","last_undegraded":"2024-04-22T01:10:35.430803+0000","last_fullsized":"2024-04-22T01:10:35.430803+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'26","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:01.444941+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:11:01.444941+0000","last_peered":"2024-04-22T01:11:01.444941+0000","last_clean":"2024-04-22T01:11:01.444941+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:11:01.444941+0000","last_undegraded":"2024-04-22T01:11:01.444941+0000","last_fullsized":"2024-04-22T01:11:01.444941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45671,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":48,"num_write":28,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176229+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:10:36.176229+0000","last_peered":"2024-04-22T01:10:36.176229+0000","last_clean":"2024-04-22T01:10:36.176229+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:10:36.176229+0000","last_undegraded":"2024-04-22T01:10:36.176229+0000","last_fullsized":"2024-04-22T01:10:36.176229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507932+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:10:35.507932+0000","last_peered":"2024-04-22T01:10:35.507932+0000","last_clean":"2024-04-22T01:10:35.507932+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:10:35.507932+0000","last_undegraded":"2024-04-22T01:10:35.507932+0000","last_fullsized":"2024-04-22T01:10:35.507932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:57.905252+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:10:57.905252+0000","last_peered":"2024-04-22T01:10:57.905252+0000","last_clean":"2024-04-22T01:10:57.905252+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:10:57.905252+0000","last_undegraded":"2024-04-22T01:10:57.905252+0000","last_fullsized":"2024-04-22T01:10:57.905252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772220+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:10:35.772220+0000","last_peered":"2024-04-22T01:10:35.772220+0000","last_clean":"2024-04-22T01:10:35.772220+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:10:35.772220+0000","last_undegraded":"2024-04-22T01:10:35.772220+0000","last_fullsized":"2024-04-22T01:10:35.772220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175941+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:10:36.175941+0000","last_peered":"2024-04-22T01:10:36.175941+0000","last_clean":"2024-04-22T01:10:36.175941+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:10:36.175941+0000","last_undegraded":"2024-04-22T01:10:36.175941+0000","last_fullsized":"2024-04-22T01:10:36.175941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891756+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:10:56.891756+0000","last_peered":"2024-04-22T01:10:56.891756+0000","last_clean":"2024-04-22T01:10:56.891756+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:10:56.891756+0000","last_undegraded":"2024-04-22T01:10:56.891756+0000","last_fullsized":"2024-04-22T01:10:56.891756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429518+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:10:35.429518+0000","last_peered":"2024-04-22T01:10:35.429518+0000","last_clean":"2024-04-22T01:10:35.429518+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:10:35.429518+0000","last_undegraded":"2024-04-22T01:10:35.429518+0000","last_fullsized":"2024-04-22T01:10:35.429518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431305+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:10:35.431305+0000","last_peered":"2024-04-22T01:10:35.431305+0000","last_clean":"2024-04-22T01:10:35.431305+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:10:35.431305+0000","last_undegraded":"2024-04-22T01:10:35.431305+0000","last_fullsized":"2024-04-22T01:10:35.431305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:00.441057+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:11:00.441057+0000","last_peered":"2024-04-22T01:11:00.441057+0000","last_clean":"2024-04-22T01:11:00.441057+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:11:00.441057+0000","last_undegraded":"2024-04-22T01:11:00.441057+0000","last_fullsized":"2024-04-22T01:11:00.441057+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429263+0000","last_change":"2024-04-22T01:10:30.462862+0000","last_active":"2024-04-22T01:10:35.429263+0000","last_peered":"2024-04-22T01:10:35.429263+0000","last_clean":"2024-04-22T01:10:35.429263+0000","last_became_active":"2024-04-22T01:10:30.462564+0000","last_became_peered":"2024-04-22T01:10:30.462564+0000","last_unstale":"2024-04-22T01:10:35.429263+0000","last_undegraded":"2024-04-22T01:10:35.429263+0000","last_fullsized":"2024-04-22T01:10:35.429263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:14:13.528582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772589+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:10:35.772589+0000","last_peered":"2024-04-22T01:10:35.772589+0000","last_clean":"2024-04-22T01:10:35.772589+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:10:35.772589+0000","last_undegraded":"2024-04-22T01:10:35.772589+0000","last_fullsized":"2024-04-22T01:10:35.772589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:57.906694+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:10:57.906694+0000","last_peered":"2024-04-22T01:10:57.906694+0000","last_clean":"2024-04-22T01:10:57.906694+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:10:57.906694+0000","last_undegraded":"2024-04-22T01:10:57.906694+0000","last_fullsized":"2024-04-22T01:10:57.906694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507268+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:10:35.507268+0000","last_peered":"2024-04-22T01:10:35.507268+0000","last_clean":"2024-04-22T01:10:35.507268+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:10:35.507268+0000","last_undegraded":"2024-04-22T01:10:35.507268+0000","last_fullsized":"2024-04-22T01:10:35.507268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507279+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:10:35.507279+0000","last_peered":"2024-04-22T01:10:35.507279+0000","last_clean":"2024-04-22T01:10:35.507279+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:10:35.507279+0000","last_undegraded":"2024-04-22T01:10:35.507279+0000","last_fullsized":"2024-04-22T01:10:35.507279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176488+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:10:36.176488+0000","last_peered":"2024-04-22T01:10:36.176488+0000","last_clean":"2024-04-22T01:10:36.176488+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:10:36.176488+0000","last_undegraded":"2024-04-22T01:10:36.176488+0000","last_fullsized":"2024-04-22T01:10:36.176488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176138+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:10:36.176138+0000","last_peered":"2024-04-22T01:10:36.176138+0000","last_clean":"2024-04-22T01:10:36.176138+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:10:36.176138+0000","last_undegraded":"2024-04-22T01:10:36.176138+0000","last_fullsized":"2024-04-22T01:10:36.176138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175928+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:10:36.175928+0000","last_peered":"2024-04-22T01:10:36.175928+0000","last_clean":"2024-04-22T01:10:36.175928+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:10:36.175928+0000","last_undegraded":"2024-04-22T01:10:36.175928+0000","last_fullsized":"2024-04-22T01:10:36.175928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431213+0000","last_change":"2024-04-22T01:10:29.590388+0000","last_active":"2024-04-22T01:10:35.431213+0000","last_peered":"2024-04-22T01:10:35.431213+0000","last_clean":"2024-04-22T01:10:35.431213+0000","last_became_active":"2024-04-22T01:10:29.590035+0000","last_became_peered":"2024-04-22T01:10:29.590035+0000","last_unstale":"2024-04-22T01:10:35.431213+0000","last_undegraded":"2024-04-22T01:10:35.431213+0000","last_fullsized":"2024-04-22T01:10:35.431213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:33.546044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176608+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:10:36.176608+0000","last_peered":"2024-04-22T01:10:36.176608+0000","last_clean":"2024-04-22T01:10:36.176608+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:10:36.176608+0000","last_undegraded":"2024-04-22T01:10:36.176608+0000","last_fullsized":"2024-04-22T01:10:36.176608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772522+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:10:35.772522+0000","last_peered":"2024-04-22T01:10:35.772522+0000","last_clean":"2024-04-22T01:10:35.772522+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:10:35.772522+0000","last_undegraded":"2024-04-22T01:10:35.772522+0000","last_fullsized":"2024-04-22T01:10:35.772522+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429532+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:10:35.429532+0000","last_peered":"2024-04-22T01:10:35.429532+0000","last_clean":"2024-04-22T01:10:35.429532+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:10:35.429532+0000","last_undegraded":"2024-04-22T01:10:35.429532+0000","last_fullsized":"2024-04-22T01:10:35.429532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175955+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:10:36.175955+0000","last_peered":"2024-04-22T01:10:36.175955+0000","last_clean":"2024-04-22T01:10:36.175955+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:10:36.175955+0000","last_undegraded":"2024-04-22T01:10:36.175955+0000","last_fullsized":"2024-04-22T01:10:36.175955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772526+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:10:35.772526+0000","last_peered":"2024-04-22T01:10:35.772526+0000","last_clean":"2024-04-22T01:10:35.772526+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:10:35.772526+0000","last_undegraded":"2024-04-22T01:10:35.772526+0000","last_fullsized":"2024-04-22T01:10:35.772526+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507195+0000","last_change":"2024-04-22T01:10:30.457611+0000","last_active":"2024-04-22T01:10:35.507195+0000","last_peered":"2024-04-22T01:10:35.507195+0000","last_clean":"2024-04-22T01:10:35.507195+0000","last_became_active":"2024-04-22T01:10:30.457272+0000","last_became_peered":"2024-04-22T01:10:30.457272+0000","last_unstale":"2024-04-22T01:10:35.507195+0000","last_undegraded":"2024-04-22T01:10:35.507195+0000","last_fullsized":"2024-04-22T01:10:35.507195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:11.233188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772028+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:10:35.772028+0000","last_peered":"2024-04-22T01:10:35.772028+0000","last_clean":"2024-04-22T01:10:35.772028+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:10:35.772028+0000","last_undegraded":"2024-04-22T01:10:35.772028+0000","last_fullsized":"2024-04-22T01:10:35.772028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429793+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:10:35.429793+0000","last_peered":"2024-04-22T01:10:35.429793+0000","last_clean":"2024-04-22T01:10:35.429793+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:10:35.429793+0000","last_undegraded":"2024-04-22T01:10:35.429793+0000","last_fullsized":"2024-04-22T01:10:35.429793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175476+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:10:36.175476+0000","last_peered":"2024-04-22T01:10:36.175476+0000","last_clean":"2024-04-22T01:10:36.175476+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:10:36.175476+0000","last_undegraded":"2024-04-22T01:10:36.175476+0000","last_fullsized":"2024-04-22T01:10:36.175476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:00.436259+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:11:00.436259+0000","last_peered":"2024-04-22T01:11:00.436259+0000","last_clean":"2024-04-22T01:11:00.436259+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:11:00.436259+0000","last_undegraded":"2024-04-22T01:11:00.436259+0000","last_fullsized":"2024-04-22T01:11:00.436259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429293+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:10:35.429293+0000","last_peered":"2024-04-22T01:10:35.429293+0000","last_clean":"2024-04-22T01:10:35.429293+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:10:35.429293+0000","last_undegraded":"2024-04-22T01:10:35.429293+0000","last_fullsized":"2024-04-22T01:10:35.429293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507688+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:10:35.507688+0000","last_peered":"2024-04-22T01:10:35.507688+0000","last_clean":"2024-04-22T01:10:35.507688+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:10:35.507688+0000","last_undegraded":"2024-04-22T01:10:35.507688+0000","last_fullsized":"2024-04-22T01:10:35.507688+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.771961+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:10:35.771961+0000","last_peered":"2024-04-22T01:10:35.771961+0000","last_clean":"2024-04-22T01:10:35.771961+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:10:35.771961+0000","last_undegraded":"2024-04-22T01:10:35.771961+0000","last_fullsized":"2024-04-22T01:10:35.771961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429207+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:10:35.429207+0000","last_peered":"2024-04-22T01:10:35.429207+0000","last_clean":"2024-04-22T01:10:35.429207+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:10:35.429207+0000","last_undegraded":"2024-04-22T01:10:35.429207+0000","last_fullsized":"2024-04-22T01:10:35.429207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175929+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:10:36.175929+0000","last_peered":"2024-04-22T01:10:36.175929+0000","last_clean":"2024-04-22T01:10:36.175929+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:10:36.175929+0000","last_undegraded":"2024-04-22T01:10:36.175929+0000","last_fullsized":"2024-04-22T01:10:36.175929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176345+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:10:36.176345+0000","last_peered":"2024-04-22T01:10:36.176345+0000","last_clean":"2024-04-22T01:10:36.176345+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:10:36.176345+0000","last_undegraded":"2024-04-22T01:10:36.176345+0000","last_fullsized":"2024-04-22T01:10:36.176345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772681+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:10:35.772681+0000","last_peered":"2024-04-22T01:10:35.772681+0000","last_clean":"2024-04-22T01:10:35.772681+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:10:35.772681+0000","last_undegraded":"2024-04-22T01:10:35.772681+0000","last_fullsized":"2024-04-22T01:10:35.772681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175826+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:10:36.175826+0000","last_peered":"2024-04-22T01:10:36.175826+0000","last_clean":"2024-04-22T01:10:36.175826+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:10:36.175826+0000","last_undegraded":"2024-04-22T01:10:36.175826+0000","last_fullsized":"2024-04-22T01:10:36.175826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.915086+0000","last_change":"2024-04-22T01:10:29.598225+0000","last_active":"2024-04-22T01:10:58.915086+0000","last_peered":"2024-04-22T01:10:58.915086+0000","last_clean":"2024-04-22T01:10:58.915086+0000","last_became_active":"2024-04-22T01:10:29.597867+0000","last_became_peered":"2024-04-22T01:10:29.597867+0000","last_unstale":"2024-04-22T01:10:58.915086+0000","last_undegraded":"2024-04-22T01:10:58.915086+0000","last_fullsized":"2024-04-22T01:10:58.915086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:52:32.759350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176125+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:10:36.176125+0000","last_peered":"2024-04-22T01:10:36.176125+0000","last_clean":"2024-04-22T01:10:36.176125+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:10:36.176125+0000","last_undegraded":"2024-04-22T01:10:36.176125+0000","last_fullsized":"2024-04-22T01:10:36.176125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429021+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:10:35.429021+0000","last_peered":"2024-04-22T01:10:35.429021+0000","last_clean":"2024-04-22T01:10:35.429021+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:10:35.429021+0000","last_undegraded":"2024-04-22T01:10:35.429021+0000","last_fullsized":"2024-04-22T01:10:35.429021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:59.926196+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:10:59.926196+0000","last_peered":"2024-04-22T01:10:59.926196+0000","last_clean":"2024-04-22T01:10:59.926196+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:10:59.926196+0000","last_undegraded":"2024-04-22T01:10:59.926196+0000","last_fullsized":"2024-04-22T01:10:59.926196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175625+0000","last_change":"2024-04-22T01:10:30.472682+0000","last_active":"2024-04-22T01:10:36.175625+0000","last_peered":"2024-04-22T01:10:36.175625+0000","last_clean":"2024-04-22T01:10:36.175625+0000","last_became_active":"2024-04-22T01:10:30.472348+0000","last_became_peered":"2024-04-22T01:10:30.472348+0000","last_unstale":"2024-04-22T01:10:36.175625+0000","last_undegraded":"2024-04-22T01:10:36.175625+0000","last_fullsized":"2024-04-22T01:10:36.175625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:18.194268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967054+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:10:35.967054+0000","last_peered":"2024-04-22T01:10:35.967054+0000","last_clean":"2024-04-22T01:10:35.967054+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:10:35.967054+0000","last_undegraded":"2024-04-22T01:10:35.967054+0000","last_fullsized":"2024-04-22T01:10:35.967054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891733+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:10:56.891733+0000","last_peered":"2024-04-22T01:10:56.891733+0000","last_clean":"2024-04-22T01:10:56.891733+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:10:56.891733+0000","last_undegraded":"2024-04-22T01:10:56.891733+0000","last_fullsized":"2024-04-22T01:10:56.891733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175661+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:10:36.175661+0000","last_peered":"2024-04-22T01:10:36.175661+0000","last_clean":"2024-04-22T01:10:36.175661+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:10:36.175661+0000","last_undegraded":"2024-04-22T01:10:36.175661+0000","last_fullsized":"2024-04-22T01:10:36.175661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431360+0000","last_change":"2024-04-22T01:10:34.673139+0000","last_active":"2024-04-22T01:10:35.431360+0000","last_peered":"2024-04-22T01:10:35.431360+0000","last_clean":"2024-04-22T01:10:35.431360+0000","last_became_active":"2024-04-22T01:10:34.672004+0000","last_became_peered":"2024-04-22T01:10:34.672004+0000","last_unstale":"2024-04-22T01:10:35.431360+0000","last_undegraded":"2024-04-22T01:10:35.431360+0000","last_fullsized":"2024-04-22T01:10:35.431360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:08.609660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:59.925258+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:10:59.925258+0000","last_peered":"2024-04-22T01:10:59.925258+0000","last_clean":"2024-04-22T01:10:59.925258+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:10:59.925258+0000","last_undegraded":"2024-04-22T01:10:59.925258+0000","last_fullsized":"2024-04-22T01:10:59.925258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175718+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:10:36.175718+0000","last_peered":"2024-04-22T01:10:36.175718+0000","last_clean":"2024-04-22T01:10:36.175718+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:10:36.175718+0000","last_undegraded":"2024-04-22T01:10:36.175718+0000","last_fullsized":"2024-04-22T01:10:36.175718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.415911+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:10:36.415911+0000","last_peered":"2024-04-22T01:10:36.415911+0000","last_clean":"2024-04-22T01:10:36.415911+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:10:36.415911+0000","last_undegraded":"2024-04-22T01:10:36.415911+0000","last_fullsized":"2024-04-22T01:10:36.415911+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.884917+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:10:58.884917+0000","last_peered":"2024-04-22T01:10:58.884917+0000","last_clean":"2024-04-22T01:10:58.884917+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:10:58.884917+0000","last_undegraded":"2024-04-22T01:10:58.884917+0000","last_fullsized":"2024-04-22T01:10:58.884917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175894+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:10:36.175894+0000","last_peered":"2024-04-22T01:10:36.175894+0000","last_clean":"2024-04-22T01:10:36.175894+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:10:36.175894+0000","last_undegraded":"2024-04-22T01:10:36.175894+0000","last_fullsized":"2024-04-22T01:10:36.175894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967286+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:10:35.967286+0000","last_peered":"2024-04-22T01:10:35.967286+0000","last_clean":"2024-04-22T01:10:35.967286+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:10:35.967286+0000","last_undegraded":"2024-04-22T01:10:35.967286+0000","last_fullsized":"2024-04-22T01:10:35.967286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.888623+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:10:56.888623+0000","last_peered":"2024-04-22T01:10:56.888623+0000","last_clean":"2024-04-22T01:10:56.888623+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:10:56.888623+0000","last_undegraded":"2024-04-22T01:10:56.888623+0000","last_fullsized":"2024-04-22T01:10:56.888623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429645+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:10:35.429645+0000","last_peered":"2024-04-22T01:10:35.429645+0000","last_clean":"2024-04-22T01:10:35.429645+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:10:35.429645+0000","last_undegraded":"2024-04-22T01:10:35.429645+0000","last_fullsized":"2024-04-22T01:10:35.429645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175824+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:10:36.175824+0000","last_peered":"2024-04-22T01:10:36.175824+0000","last_clean":"2024-04-22T01:10:36.175824+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:10:36.175824+0000","last_undegraded":"2024-04-22T01:10:36.175824+0000","last_fullsized":"2024-04-22T01:10:36.175824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.911974+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:10:58.911974+0000","last_peered":"2024-04-22T01:10:58.911974+0000","last_clean":"2024-04-22T01:10:58.911974+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:10:58.911974+0000","last_undegraded":"2024-04-22T01:10:58.911974+0000","last_fullsized":"2024-04-22T01:10:58.911974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429894+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:10:35.429894+0000","last_peered":"2024-04-22T01:10:35.429894+0000","last_clean":"2024-04-22T01:10:35.429894+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:10:35.429894+0000","last_undegraded":"2024-04-22T01:10:35.429894+0000","last_fullsized":"2024-04-22T01:10:35.429894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175085+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:10:36.175085+0000","last_peered":"2024-04-22T01:10:36.175085+0000","last_clean":"2024-04-22T01:10:36.175085+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:10:36.175085+0000","last_undegraded":"2024-04-22T01:10:36.175085+0000","last_fullsized":"2024-04-22T01:10:36.175085+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507825+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:10:35.507825+0000","last_peered":"2024-04-22T01:10:35.507825+0000","last_clean":"2024-04-22T01:10:35.507825+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:10:35.507825+0000","last_undegraded":"2024-04-22T01:10:35.507825+0000","last_fullsized":"2024-04-22T01:10:35.507825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429386+0000","last_change":"2024-04-22T01:10:30.471548+0000","last_active":"2024-04-22T01:10:35.429386+0000","last_peered":"2024-04-22T01:10:35.429386+0000","last_clean":"2024-04-22T01:10:35.429386+0000","last_became_active":"2024-04-22T01:10:30.471219+0000","last_became_peered":"2024-04-22T01:10:30.471219+0000","last_unstale":"2024-04-22T01:10:35.429386+0000","last_undegraded":"2024-04-22T01:10:35.429386+0000","last_fullsized":"2024-04-22T01:10:35.429386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:27.776929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507030+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:10:35.507030+0000","last_peered":"2024-04-22T01:10:35.507030+0000","last_clean":"2024-04-22T01:10:35.507030+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:10:35.507030+0000","last_undegraded":"2024-04-22T01:10:35.507030+0000","last_fullsized":"2024-04-22T01:10:35.507030+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772488+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:10:35.772488+0000","last_peered":"2024-04-22T01:10:35.772488+0000","last_clean":"2024-04-22T01:10:35.772488+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:10:35.772488+0000","last_undegraded":"2024-04-22T01:10:35.772488+0000","last_fullsized":"2024-04-22T01:10:35.772488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429911+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:10:35.429911+0000","last_peered":"2024-04-22T01:10:35.429911+0000","last_clean":"2024-04-22T01:10:35.429911+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:10:35.429911+0000","last_undegraded":"2024-04-22T01:10:35.429911+0000","last_fullsized":"2024-04-22T01:10:35.429911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176034+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:10:36.176034+0000","last_peered":"2024-04-22T01:10:36.176034+0000","last_clean":"2024-04-22T01:10:36.176034+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:10:36.176034+0000","last_undegraded":"2024-04-22T01:10:36.176034+0000","last_fullsized":"2024-04-22T01:10:36.176034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429125+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:10:35.429125+0000","last_peered":"2024-04-22T01:10:35.429125+0000","last_clean":"2024-04-22T01:10:35.429125+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:10:35.429125+0000","last_undegraded":"2024-04-22T01:10:35.429125+0000","last_fullsized":"2024-04-22T01:10:35.429125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429016+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:10:35.429016+0000","last_peered":"2024-04-22T01:10:35.429016+0000","last_clean":"2024-04-22T01:10:35.429016+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:10:35.429016+0000","last_undegraded":"2024-04-22T01:10:35.429016+0000","last_fullsized":"2024-04-22T01:10:35.429016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175214+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:10:36.175214+0000","last_peered":"2024-04-22T01:10:36.175214+0000","last_clean":"2024-04-22T01:10:36.175214+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:10:36.175214+0000","last_undegraded":"2024-04-22T01:10:36.175214+0000","last_fullsized":"2024-04-22T01:10:36.175214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772601+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:10:35.772601+0000","last_peered":"2024-04-22T01:10:35.772601+0000","last_clean":"2024-04-22T01:10:35.772601+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:10:35.772601+0000","last_undegraded":"2024-04-22T01:10:35.772601+0000","last_fullsized":"2024-04-22T01:10:35.772601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176661+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:10:36.176661+0000","last_peered":"2024-04-22T01:10:36.176661+0000","last_clean":"2024-04-22T01:10:36.176661+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:10:36.176661+0000","last_undegraded":"2024-04-22T01:10:36.176661+0000","last_fullsized":"2024-04-22T01:10:36.176661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429670+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:10:35.429670+0000","last_peered":"2024-04-22T01:10:35.429670+0000","last_clean":"2024-04-22T01:10:35.429670+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:10:35.429670+0000","last_undegraded":"2024-04-22T01:10:35.429670+0000","last_fullsized":"2024-04-22T01:10:35.429670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176275+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:10:36.176275+0000","last_peered":"2024-04-22T01:10:36.176275+0000","last_clean":"2024-04-22T01:10:36.176275+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:10:36.176275+0000","last_undegraded":"2024-04-22T01:10:36.176275+0000","last_fullsized":"2024-04-22T01:10:36.176275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507188+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:10:35.507188+0000","last_peered":"2024-04-22T01:10:35.507188+0000","last_clean":"2024-04-22T01:10:35.507188+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:10:35.507188+0000","last_undegraded":"2024-04-22T01:10:35.507188+0000","last_fullsized":"2024-04-22T01:10:35.507188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176375+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:10:36.176375+0000","last_peered":"2024-04-22T01:10:36.176375+0000","last_clean":"2024-04-22T01:10:36.176375+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:10:36.176375+0000","last_undegraded":"2024-04-22T01:10:36.176375+0000","last_fullsized":"2024-04-22T01:10:36.176375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176199+0000","last_change":"2024-04-22T01:10:30.477852+0000","last_active":"2024-04-22T01:10:36.176199+0000","last_peered":"2024-04-22T01:10:36.176199+0000","last_clean":"2024-04-22T01:10:36.176199+0000","last_became_active":"2024-04-22T01:10:30.477606+0000","last_became_peered":"2024-04-22T01:10:30.477606+0000","last_unstale":"2024-04-22T01:10:36.176199+0000","last_undegraded":"2024-04-22T01:10:36.176199+0000","last_fullsized":"2024-04-22T01:10:36.176199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:21:39.983985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772305+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:10:35.772305+0000","last_peered":"2024-04-22T01:10:35.772305+0000","last_clean":"2024-04-22T01:10:35.772305+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:10:35.772305+0000","last_undegraded":"2024-04-22T01:10:35.772305+0000","last_fullsized":"2024-04-22T01:10:35.772305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176240+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:10:36.176240+0000","last_peered":"2024-04-22T01:10:36.176240+0000","last_clean":"2024-04-22T01:10:36.176240+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:10:36.176240+0000","last_undegraded":"2024-04-22T01:10:36.176240+0000","last_fullsized":"2024-04-22T01:10:36.176240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":51112,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":23,"num_read_kb":63,"num_write":176,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":73520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":478407,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1109790,"data_compressed":51092,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":482433,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":51092,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":489409,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477623,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920824,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477915,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":51092,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":12194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":47407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:11:02.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:11:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:01 smithi111 ceph-mon[31057]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:01 smithi111 ceph-mon[31057]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:11:02.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:01 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3906527887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:11:02.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 -- 172.21.15.38:0/1123858485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0102980 msgr2=0x7f7af0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 --2- 172.21.15.38:0/1123858485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0102980 0x7f7af0102e00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009900 tx=0x7f7ae002bc30 comp rx=0 tx=0).stop 2024-04-22T01:11:02.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 -- 172.21.15.38:0/1123858485 shutdown_connections 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 --2- 172.21.15.38:0/1123858485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7af0103340 0x7f7af0109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 --2- 172.21.15.38:0/1123858485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0102980 0x7f7af0102e00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 --2- 172.21.15.38:0/1123858485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 0x7f7af0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 -- 172.21.15.38:0/1123858485 >> 172.21.15.38:0/1123858485 conn(0x7f7af00fcd20 msgr2=0x7f7af00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 -- 172.21.15.38:0/1123858485 shutdown_connections 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.529+0000 7f7af7e20640 1 -- 172.21.15.38:0/1123858485 wait complete. 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 Processor -- start 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 -- start start 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 0x7f7af0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7af0102980 0x7f7af0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 0x7f7af019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010d3f0 con 0x7f7af0103340 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010d270 con 0x7f7af0101790 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af7e20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010d570 con 0x7f7af0102980 2024-04-22T01:11:02.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 0x7f7af019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 0x7f7af019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44520/0 (socket says 172.21.15.38:44520) 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 -- 172.21.15.38:0/271474125 learned_addr learned my addr 172.21.15.38:0/271474125 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af53bb640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7af0102980 0x7f7af0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 -- 172.21.15.38:0/271474125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7af0102980 msgr2=0x7f7af0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7af0102980 0x7f7af0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af5bbc640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 0x7f7af0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 -- 172.21.15.38:0/271474125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 msgr2=0x7f7af0197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 0x7f7af0197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af63bd640 1 -- 172.21.15.38:0/271474125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af019f6c0 con 0x7f7af0103340 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.530+0000 7f7af5bbc640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 0x7f7af0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.531+0000 7f7af63bd640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 0x7f7af019f050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800cce0 tx=0x7f7ae800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.531+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae800f450 con 0x7f7af0103340 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.531+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae800fad0 con 0x7f7af0103340 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.531+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af019f9b0 con 0x7f7af0103340 2024-04-22T01:11:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.531+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8018c10 con 0x7f7af0103340 2024-04-22T01:11:02.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.531+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af019feb0 con 0x7f7af0103340 2024-04-22T01:11:02.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.532+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af0108a50 con 0x7f7af0103340 2024-04-22T01:11:02.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.533+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7ae800f5f0 con 0x7f7af0103340 2024-04-22T01:11:02.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.533+0000 7f7ae6ffd640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7adc0a2c20 0x7f7adc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:02.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.534+0000 7f7af5bbc640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7adc0a2c20 0x7f7adc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:02.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.534+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7ae80cfc10 con 0x7f7af0103340 2024-04-22T01:11:02.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.534+0000 7f7af5bbc640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7adc0a2c20 0x7f7adc0a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7af01027e0 tx=0x7f7aec005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:02.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.537+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7ae8014030 con 0x7f7af0103340 2024-04-22T01:11:02.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.718+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7af0061b80 con 0x7f7adc0a2c20 2024-04-22T01:11:02.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.727+0000 7f7ae6ffd640 1 -- 172.21.15.38:0/271474125 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340633 (secure 0 0 0) 0x7f7af0061b80 con 0x7f7adc0a2c20 2024-04-22T01:11:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:02.735 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7adc0a2c20 msgr2=0x7f7adc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7adc0a2c20 0x7f7adc0a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7af01027e0 tx=0x7f7aec005e70 comp rx=0 tx=0).stop 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 msgr2=0x7f7af019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 0x7f7af019f050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800cce0 tx=0x7f7ae800a620 comp rx=0 tx=0).stop 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 shutdown_connections 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7adc0a2c20 0x7f7adc0a50e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7af0103340 0x7f7af019f050 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7af0102980 0x7f7af0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 --2- 172.21.15.38:0/271474125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7af0101790 0x7f7af0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:02.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 >> 172.21.15.38:0/271474125 conn(0x7f7af00fcd20 msgr2=0x7f7af0105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:02.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 shutdown_connections 2024-04-22T01:11:02.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:02.731+0000 7f7af7e20640 1 -- 172.21.15.38:0/271474125 wait complete. 2024-04-22T01:11:02.743 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-04-22T01:11:01.790535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":641597,"num_objects":78,"num_object_clones":0,"num_object_copies":235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":149,"num_read_kb":172,"num_write":358,"num_write_kb":3043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":292,"ondisk_log_size":292,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356736,"kb_used_data":11736,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585088,"statfs":{"total":1151940427776,"available":1151575130112,"internally_reserved":0,"allocated":12017664,"data_stored":7567681,"data_compressed":285387,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46730,"num_objects":54,"num_object_clones":0,"num_object_copies":163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":23,"num_read_kb":63,"num_write":130,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003909"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176576+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:10:36.176576+0000","last_peered":"2024-04-22T01:10:36.176576+0000","last_clean":"2024-04-22T01:10:36.176576+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:10:36.176576+0000","last_undegraded":"2024-04-22T01:10:36.176576+0000","last_fullsized":"2024-04-22T01:10:36.176576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175529+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:10:36.175529+0000","last_peered":"2024-04-22T01:10:36.175529+0000","last_clean":"2024-04-22T01:10:36.175529+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:10:36.175529+0000","last_undegraded":"2024-04-22T01:10:36.175529+0000","last_fullsized":"2024-04-22T01:10:36.175529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175264+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:10:36.175264+0000","last_peered":"2024-04-22T01:10:36.175264+0000","last_clean":"2024-04-22T01:10:36.175264+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:10:36.175264+0000","last_undegraded":"2024-04-22T01:10:36.175264+0000","last_fullsized":"2024-04-22T01:10:36.175264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967267+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:10:35.967267+0000","last_peered":"2024-04-22T01:10:35.967267+0000","last_clean":"2024-04-22T01:10:35.967267+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:10:35.967267+0000","last_undegraded":"2024-04-22T01:10:35.967267+0000","last_fullsized":"2024-04-22T01:10:35.967267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175960+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:10:36.175960+0000","last_peered":"2024-04-22T01:10:36.175960+0000","last_clean":"2024-04-22T01:10:36.175960+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:10:36.175960+0000","last_undegraded":"2024-04-22T01:10:36.175960+0000","last_fullsized":"2024-04-22T01:10:36.175960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175929+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:10:36.175929+0000","last_peered":"2024-04-22T01:10:36.175929+0000","last_clean":"2024-04-22T01:10:36.175929+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:10:36.175929+0000","last_undegraded":"2024-04-22T01:10:36.175929+0000","last_fullsized":"2024-04-22T01:10:36.175929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429231+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:10:35.429231+0000","last_peered":"2024-04-22T01:10:35.429231+0000","last_clean":"2024-04-22T01:10:35.429231+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:10:35.429231+0000","last_undegraded":"2024-04-22T01:10:35.429231+0000","last_fullsized":"2024-04-22T01:10:35.429231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175589+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:10:36.175589+0000","last_peered":"2024-04-22T01:10:36.175589+0000","last_clean":"2024-04-22T01:10:36.175589+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:10:36.175589+0000","last_undegraded":"2024-04-22T01:10:36.175589+0000","last_fullsized":"2024-04-22T01:10:36.175589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176623+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:10:36.176623+0000","last_peered":"2024-04-22T01:10:36.176623+0000","last_clean":"2024-04-22T01:10:36.176623+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:10:36.176623+0000","last_undegraded":"2024-04-22T01:10:36.176623+0000","last_fullsized":"2024-04-22T01:10:36.176623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176540+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:10:36.176540+0000","last_peered":"2024-04-22T01:10:36.176540+0000","last_clean":"2024-04-22T01:10:36.176540+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:10:36.176540+0000","last_undegraded":"2024-04-22T01:10:36.176540+0000","last_fullsized":"2024-04-22T01:10:36.176540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429879+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:10:35.429879+0000","last_peered":"2024-04-22T01:10:35.429879+0000","last_clean":"2024-04-22T01:10:35.429879+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:10:35.429879+0000","last_undegraded":"2024-04-22T01:10:35.429879+0000","last_fullsized":"2024-04-22T01:10:35.429879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967105+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:10:35.967105+0000","last_peered":"2024-04-22T01:10:35.967105+0000","last_clean":"2024-04-22T01:10:35.967105+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:10:35.967105+0000","last_undegraded":"2024-04-22T01:10:35.967105+0000","last_fullsized":"2024-04-22T01:10:35.967105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772199+0000","last_change":"2024-04-22T01:10:30.452385+0000","last_active":"2024-04-22T01:10:35.772199+0000","last_peered":"2024-04-22T01:10:35.772199+0000","last_clean":"2024-04-22T01:10:35.772199+0000","last_became_active":"2024-04-22T01:10:30.452034+0000","last_became_peered":"2024-04-22T01:10:30.452034+0000","last_unstale":"2024-04-22T01:10:35.772199+0000","last_undegraded":"2024-04-22T01:10:35.772199+0000","last_fullsized":"2024-04-22T01:10:35.772199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:14:02.046478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.174949+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:10:36.174949+0000","last_peered":"2024-04-22T01:10:36.174949+0000","last_clean":"2024-04-22T01:10:36.174949+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:10:36.174949+0000","last_undegraded":"2024-04-22T01:10:36.174949+0000","last_fullsized":"2024-04-22T01:10:36.174949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.174991+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:10:36.174991+0000","last_peered":"2024-04-22T01:10:36.174991+0000","last_clean":"2024-04-22T01:10:36.174991+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:10:36.174991+0000","last_undegraded":"2024-04-22T01:10:36.174991+0000","last_fullsized":"2024-04-22T01:10:36.174991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175718+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:10:36.175718+0000","last_peered":"2024-04-22T01:10:36.175718+0000","last_clean":"2024-04-22T01:10:36.175718+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:10:36.175718+0000","last_undegraded":"2024-04-22T01:10:36.175718+0000","last_fullsized":"2024-04-22T01:10:36.175718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176239+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:10:36.176239+0000","last_peered":"2024-04-22T01:10:36.176239+0000","last_clean":"2024-04-22T01:10:36.176239+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:10:36.176239+0000","last_undegraded":"2024-04-22T01:10:36.176239+0000","last_fullsized":"2024-04-22T01:10:36.176239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431380+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:10:35.431380+0000","last_peered":"2024-04-22T01:10:35.431380+0000","last_clean":"2024-04-22T01:10:35.431380+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:10:35.431380+0000","last_undegraded":"2024-04-22T01:10:35.431380+0000","last_fullsized":"2024-04-22T01:10:35.431380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429794+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:10:35.429794+0000","last_peered":"2024-04-22T01:10:35.429794+0000","last_clean":"2024-04-22T01:10:35.429794+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:10:35.429794+0000","last_undegraded":"2024-04-22T01:10:35.429794+0000","last_fullsized":"2024-04-22T01:10:35.429794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429385+0000","last_change":"2024-04-22T01:10:34.667483+0000","last_active":"2024-04-22T01:10:35.429385+0000","last_peered":"2024-04-22T01:10:35.429385+0000","last_clean":"2024-04-22T01:10:35.429385+0000","last_became_active":"2024-04-22T01:10:34.666997+0000","last_became_peered":"2024-04-22T01:10:34.666997+0000","last_unstale":"2024-04-22T01:10:35.429385+0000","last_undegraded":"2024-04-22T01:10:35.429385+0000","last_fullsized":"2024-04-22T01:10:35.429385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:03:35.317553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:00.441670+0000","last_change":"2024-04-22T01:10:29.585069+0000","last_active":"2024-04-22T01:11:00.441670+0000","last_peered":"2024-04-22T01:11:00.441670+0000","last_clean":"2024-04-22T01:11:00.441670+0000","last_became_active":"2024-04-22T01:10:29.584651+0000","last_became_peered":"2024-04-22T01:10:29.584651+0000","last_unstale":"2024-04-22T01:11:00.441670+0000","last_undegraded":"2024-04-22T01:11:00.441670+0000","last_fullsized":"2024-04-22T01:11:00.441670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:13.121214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176127+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:10:36.176127+0000","last_peered":"2024-04-22T01:10:36.176127+0000","last_clean":"2024-04-22T01:10:36.176127+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:10:36.176127+0000","last_undegraded":"2024-04-22T01:10:36.176127+0000","last_fullsized":"2024-04-22T01:10:36.176127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.966796+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:10:35.966796+0000","last_peered":"2024-04-22T01:10:35.966796+0000","last_clean":"2024-04-22T01:10:35.966796+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:10:35.966796+0000","last_undegraded":"2024-04-22T01:10:35.966796+0000","last_fullsized":"2024-04-22T01:10:35.966796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176561+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:10:36.176561+0000","last_peered":"2024-04-22T01:10:36.176561+0000","last_clean":"2024-04-22T01:10:36.176561+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:10:36.176561+0000","last_undegraded":"2024-04-22T01:10:36.176561+0000","last_fullsized":"2024-04-22T01:10:36.176561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175408+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:10:36.175408+0000","last_peered":"2024-04-22T01:10:36.175408+0000","last_clean":"2024-04-22T01:10:36.175408+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:10:36.175408+0000","last_undegraded":"2024-04-22T01:10:36.175408+0000","last_fullsized":"2024-04-22T01:10:36.175408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.966823+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:10:35.966823+0000","last_peered":"2024-04-22T01:10:35.966823+0000","last_clean":"2024-04-22T01:10:35.966823+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:10:35.966823+0000","last_undegraded":"2024-04-22T01:10:35.966823+0000","last_fullsized":"2024-04-22T01:10:35.966823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:57.905051+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:10:57.905051+0000","last_peered":"2024-04-22T01:10:57.905051+0000","last_clean":"2024-04-22T01:10:57.905051+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:10:57.905051+0000","last_undegraded":"2024-04-22T01:10:57.905051+0000","last_fullsized":"2024-04-22T01:10:57.905051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176403+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:10:36.176403+0000","last_peered":"2024-04-22T01:10:36.176403+0000","last_clean":"2024-04-22T01:10:36.176403+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:10:36.176403+0000","last_undegraded":"2024-04-22T01:10:36.176403+0000","last_fullsized":"2024-04-22T01:10:36.176403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.966821+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:10:35.966821+0000","last_peered":"2024-04-22T01:10:35.966821+0000","last_clean":"2024-04-22T01:10:35.966821+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:10:35.966821+0000","last_undegraded":"2024-04-22T01:10:35.966821+0000","last_fullsized":"2024-04-22T01:10:35.966821+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176322+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:10:36.176322+0000","last_peered":"2024-04-22T01:10:36.176322+0000","last_clean":"2024-04-22T01:10:36.176322+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:10:36.176322+0000","last_undegraded":"2024-04-22T01:10:36.176322+0000","last_fullsized":"2024-04-22T01:10:36.176322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507530+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:10:35.507530+0000","last_peered":"2024-04-22T01:10:35.507530+0000","last_clean":"2024-04-22T01:10:35.507530+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:10:35.507530+0000","last_undegraded":"2024-04-22T01:10:35.507530+0000","last_fullsized":"2024-04-22T01:10:35.507530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429495+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:10:35.429495+0000","last_peered":"2024-04-22T01:10:35.429495+0000","last_clean":"2024-04-22T01:10:35.429495+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:10:35.429495+0000","last_undegraded":"2024-04-22T01:10:35.429495+0000","last_fullsized":"2024-04-22T01:10:35.429495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.893426+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:10:56.893426+0000","last_peered":"2024-04-22T01:10:56.893426+0000","last_clean":"2024-04-22T01:10:56.893426+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:10:56.893426+0000","last_undegraded":"2024-04-22T01:10:56.893426+0000","last_fullsized":"2024-04-22T01:10:56.893426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175921+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:10:36.175921+0000","last_peered":"2024-04-22T01:10:36.175921+0000","last_clean":"2024-04-22T01:10:36.175921+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:10:36.175921+0000","last_undegraded":"2024-04-22T01:10:36.175921+0000","last_fullsized":"2024-04-22T01:10:36.175921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429594+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:10:35.429594+0000","last_peered":"2024-04-22T01:10:35.429594+0000","last_clean":"2024-04-22T01:10:35.429594+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:10:35.429594+0000","last_undegraded":"2024-04-22T01:10:35.429594+0000","last_fullsized":"2024-04-22T01:10:35.429594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.914913+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:10:58.914913+0000","last_peered":"2024-04-22T01:10:58.914913+0000","last_clean":"2024-04-22T01:10:58.914913+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:10:58.914913+0000","last_undegraded":"2024-04-22T01:10:58.914913+0000","last_fullsized":"2024-04-22T01:10:58.914913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507706+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:10:35.507706+0000","last_peered":"2024-04-22T01:10:35.507706+0000","last_clean":"2024-04-22T01:10:35.507706+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:10:35.507706+0000","last_undegraded":"2024-04-22T01:10:35.507706+0000","last_fullsized":"2024-04-22T01:10:35.507706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176059+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:10:36.176059+0000","last_peered":"2024-04-22T01:10:36.176059+0000","last_clean":"2024-04-22T01:10:36.176059+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:10:36.176059+0000","last_undegraded":"2024-04-22T01:10:36.176059+0000","last_fullsized":"2024-04-22T01:10:36.176059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772110+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:10:35.772110+0000","last_peered":"2024-04-22T01:10:35.772110+0000","last_clean":"2024-04-22T01:10:35.772110+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:10:35.772110+0000","last_undegraded":"2024-04-22T01:10:35.772110+0000","last_fullsized":"2024-04-22T01:10:35.772110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175949+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:10:36.175949+0000","last_peered":"2024-04-22T01:10:36.175949+0000","last_clean":"2024-04-22T01:10:36.175949+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:10:36.175949+0000","last_undegraded":"2024-04-22T01:10:36.175949+0000","last_fullsized":"2024-04-22T01:10:36.175949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175475+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:10:36.175475+0000","last_peered":"2024-04-22T01:10:36.175475+0000","last_clean":"2024-04-22T01:10:36.175475+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:10:36.175475+0000","last_undegraded":"2024-04-22T01:10:36.175475+0000","last_fullsized":"2024-04-22T01:10:36.175475+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891531+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:10:56.891531+0000","last_peered":"2024-04-22T01:10:56.891531+0000","last_clean":"2024-04-22T01:10:56.891531+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:10:56.891531+0000","last_undegraded":"2024-04-22T01:10:56.891531+0000","last_fullsized":"2024-04-22T01:10:56.891531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.430603+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:10:35.430603+0000","last_peered":"2024-04-22T01:10:35.430603+0000","last_clean":"2024-04-22T01:10:35.430603+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:10:35.430603+0000","last_undegraded":"2024-04-22T01:10:35.430603+0000","last_fullsized":"2024-04-22T01:10:35.430603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176153+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:10:36.176153+0000","last_peered":"2024-04-22T01:10:36.176153+0000","last_clean":"2024-04-22T01:10:36.176153+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:10:36.176153+0000","last_undegraded":"2024-04-22T01:10:36.176153+0000","last_fullsized":"2024-04-22T01:10:36.176153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:01.440127+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:11:01.440127+0000","last_peered":"2024-04-22T01:11:01.440127+0000","last_clean":"2024-04-22T01:11:01.440127+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:11:01.440127+0000","last_undegraded":"2024-04-22T01:11:01.440127+0000","last_fullsized":"2024-04-22T01:11:01.440127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176183+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:10:36.176183+0000","last_peered":"2024-04-22T01:10:36.176183+0000","last_clean":"2024-04-22T01:10:36.176183+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:10:36.176183+0000","last_undegraded":"2024-04-22T01:10:36.176183+0000","last_fullsized":"2024-04-22T01:10:36.176183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.430820+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:10:35.430820+0000","last_peered":"2024-04-22T01:10:35.430820+0000","last_clean":"2024-04-22T01:10:35.430820+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:10:35.430820+0000","last_undegraded":"2024-04-22T01:10:35.430820+0000","last_fullsized":"2024-04-22T01:10:35.430820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891351+0000","last_change":"2024-04-22T01:10:29.591793+0000","last_active":"2024-04-22T01:10:56.891351+0000","last_peered":"2024-04-22T01:10:56.891351+0000","last_clean":"2024-04-22T01:10:56.891351+0000","last_became_active":"2024-04-22T01:10:29.591245+0000","last_became_peered":"2024-04-22T01:10:29.591245+0000","last_unstale":"2024-04-22T01:10:56.891351+0000","last_undegraded":"2024-04-22T01:10:56.891351+0000","last_fullsized":"2024-04-22T01:10:56.891351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:40:28.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175366+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:10:36.175366+0000","last_peered":"2024-04-22T01:10:36.175366+0000","last_clean":"2024-04-22T01:10:36.175366+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:10:36.175366+0000","last_undegraded":"2024-04-22T01:10:36.175366+0000","last_fullsized":"2024-04-22T01:10:36.175366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967313+0000","last_change":"2024-04-22T01:10:34.671540+0000","last_active":"2024-04-22T01:10:35.967313+0000","last_peered":"2024-04-22T01:10:35.967313+0000","last_clean":"2024-04-22T01:10:35.967313+0000","last_became_active":"2024-04-22T01:10:34.670777+0000","last_became_peered":"2024-04-22T01:10:34.670777+0000","last_unstale":"2024-04-22T01:10:35.967313+0000","last_undegraded":"2024-04-22T01:10:35.967313+0000","last_fullsized":"2024-04-22T01:10:35.967313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:39:32.654313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.890841+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:10:56.890841+0000","last_peered":"2024-04-22T01:10:56.890841+0000","last_clean":"2024-04-22T01:10:56.890841+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:10:56.890841+0000","last_undegraded":"2024-04-22T01:10:56.890841+0000","last_fullsized":"2024-04-22T01:10:56.890841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176492+0000","last_change":"2024-04-22T01:10:30.473144+0000","last_active":"2024-04-22T01:10:36.176492+0000","last_peered":"2024-04-22T01:10:36.176492+0000","last_clean":"2024-04-22T01:10:36.176492+0000","last_became_active":"2024-04-22T01:10:30.472860+0000","last_became_peered":"2024-04-22T01:10:30.472860+0000","last_unstale":"2024-04-22T01:10:36.176492+0000","last_undegraded":"2024-04-22T01:10:36.176492+0000","last_fullsized":"2024-04-22T01:10:36.176492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:14.097318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.430803+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:10:35.430803+0000","last_peered":"2024-04-22T01:10:35.430803+0000","last_clean":"2024-04-22T01:10:35.430803+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:10:35.430803+0000","last_undegraded":"2024-04-22T01:10:35.430803+0000","last_fullsized":"2024-04-22T01:10:35.430803+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'26","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:01.444941+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:11:01.444941+0000","last_peered":"2024-04-22T01:11:01.444941+0000","last_clean":"2024-04-22T01:11:01.444941+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:11:01.444941+0000","last_undegraded":"2024-04-22T01:11:01.444941+0000","last_fullsized":"2024-04-22T01:11:01.444941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45671,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":48,"num_write":28,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176229+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:10:36.176229+0000","last_peered":"2024-04-22T01:10:36.176229+0000","last_clean":"2024-04-22T01:10:36.176229+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:10:36.176229+0000","last_undegraded":"2024-04-22T01:10:36.176229+0000","last_fullsized":"2024-04-22T01:10:36.176229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507932+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:10:35.507932+0000","last_peered":"2024-04-22T01:10:35.507932+0000","last_clean":"2024-04-22T01:10:35.507932+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:10:35.507932+0000","last_undegraded":"2024-04-22T01:10:35.507932+0000","last_fullsized":"2024-04-22T01:10:35.507932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:57.905252+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:10:57.905252+0000","last_peered":"2024-04-22T01:10:57.905252+0000","last_clean":"2024-04-22T01:10:57.905252+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:10:57.905252+0000","last_undegraded":"2024-04-22T01:10:57.905252+0000","last_fullsized":"2024-04-22T01:10:57.905252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772220+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:10:35.772220+0000","last_peered":"2024-04-22T01:10:35.772220+0000","last_clean":"2024-04-22T01:10:35.772220+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:10:35.772220+0000","last_undegraded":"2024-04-22T01:10:35.772220+0000","last_fullsized":"2024-04-22T01:10:35.772220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175941+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:10:36.175941+0000","last_peered":"2024-04-22T01:10:36.175941+0000","last_clean":"2024-04-22T01:10:36.175941+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:10:36.175941+0000","last_undegraded":"2024-04-22T01:10:36.175941+0000","last_fullsized":"2024-04-22T01:10:36.175941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891756+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:10:56.891756+0000","last_peered":"2024-04-22T01:10:56.891756+0000","last_clean":"2024-04-22T01:10:56.891756+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:10:56.891756+0000","last_undegraded":"2024-04-22T01:10:56.891756+0000","last_fullsized":"2024-04-22T01:10:56.891756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429518+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:10:35.429518+0000","last_peered":"2024-04-22T01:10:35.429518+0000","last_clean":"2024-04-22T01:10:35.429518+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:10:35.429518+0000","last_undegraded":"2024-04-22T01:10:35.429518+0000","last_fullsized":"2024-04-22T01:10:35.429518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431305+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:10:35.431305+0000","last_peered":"2024-04-22T01:10:35.431305+0000","last_clean":"2024-04-22T01:10:35.431305+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:10:35.431305+0000","last_undegraded":"2024-04-22T01:10:35.431305+0000","last_fullsized":"2024-04-22T01:10:35.431305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:00.441057+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:11:00.441057+0000","last_peered":"2024-04-22T01:11:00.441057+0000","last_clean":"2024-04-22T01:11:00.441057+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:11:00.441057+0000","last_undegraded":"2024-04-22T01:11:00.441057+0000","last_fullsized":"2024-04-22T01:11:00.441057+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429263+0000","last_change":"2024-04-22T01:10:30.462862+0000","last_active":"2024-04-22T01:10:35.429263+0000","last_peered":"2024-04-22T01:10:35.429263+0000","last_clean":"2024-04-22T01:10:35.429263+0000","last_became_active":"2024-04-22T01:10:30.462564+0000","last_became_peered":"2024-04-22T01:10:30.462564+0000","last_unstale":"2024-04-22T01:10:35.429263+0000","last_undegraded":"2024-04-22T01:10:35.429263+0000","last_fullsized":"2024-04-22T01:10:35.429263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:14:13.528582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772589+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:10:35.772589+0000","last_peered":"2024-04-22T01:10:35.772589+0000","last_clean":"2024-04-22T01:10:35.772589+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:10:35.772589+0000","last_undegraded":"2024-04-22T01:10:35.772589+0000","last_fullsized":"2024-04-22T01:10:35.772589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:57.906694+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:10:57.906694+0000","last_peered":"2024-04-22T01:10:57.906694+0000","last_clean":"2024-04-22T01:10:57.906694+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:10:57.906694+0000","last_undegraded":"2024-04-22T01:10:57.906694+0000","last_fullsized":"2024-04-22T01:10:57.906694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507268+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:10:35.507268+0000","last_peered":"2024-04-22T01:10:35.507268+0000","last_clean":"2024-04-22T01:10:35.507268+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:10:35.507268+0000","last_undegraded":"2024-04-22T01:10:35.507268+0000","last_fullsized":"2024-04-22T01:10:35.507268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507279+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:10:35.507279+0000","last_peered":"2024-04-22T01:10:35.507279+0000","last_clean":"2024-04-22T01:10:35.507279+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:10:35.507279+0000","last_undegraded":"2024-04-22T01:10:35.507279+0000","last_fullsized":"2024-04-22T01:10:35.507279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176488+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:10:36.176488+0000","last_peered":"2024-04-22T01:10:36.176488+0000","last_clean":"2024-04-22T01:10:36.176488+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:10:36.176488+0000","last_undegraded":"2024-04-22T01:10:36.176488+0000","last_fullsized":"2024-04-22T01:10:36.176488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176138+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:10:36.176138+0000","last_peered":"2024-04-22T01:10:36.176138+0000","last_clean":"2024-04-22T01:10:36.176138+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:10:36.176138+0000","last_undegraded":"2024-04-22T01:10:36.176138+0000","last_fullsized":"2024-04-22T01:10:36.176138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175928+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:10:36.175928+0000","last_peered":"2024-04-22T01:10:36.175928+0000","last_clean":"2024-04-22T01:10:36.175928+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:10:36.175928+0000","last_undegraded":"2024-04-22T01:10:36.175928+0000","last_fullsized":"2024-04-22T01:10:36.175928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431213+0000","last_change":"2024-04-22T01:10:29.590388+0000","last_active":"2024-04-22T01:10:35.431213+0000","last_peered":"2024-04-22T01:10:35.431213+0000","last_clean":"2024-04-22T01:10:35.431213+0000","last_became_active":"2024-04-22T01:10:29.590035+0000","last_became_peered":"2024-04-22T01:10:29.590035+0000","last_unstale":"2024-04-22T01:10:35.431213+0000","last_undegraded":"2024-04-22T01:10:35.431213+0000","last_fullsized":"2024-04-22T01:10:35.431213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:33.546044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176608+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:10:36.176608+0000","last_peered":"2024-04-22T01:10:36.176608+0000","last_clean":"2024-04-22T01:10:36.176608+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:10:36.176608+0000","last_undegraded":"2024-04-22T01:10:36.176608+0000","last_fullsized":"2024-04-22T01:10:36.176608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772522+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:10:35.772522+0000","last_peered":"2024-04-22T01:10:35.772522+0000","last_clean":"2024-04-22T01:10:35.772522+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:10:35.772522+0000","last_undegraded":"2024-04-22T01:10:35.772522+0000","last_fullsized":"2024-04-22T01:10:35.772522+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429532+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:10:35.429532+0000","last_peered":"2024-04-22T01:10:35.429532+0000","last_clean":"2024-04-22T01:10:35.429532+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:10:35.429532+0000","last_undegraded":"2024-04-22T01:10:35.429532+0000","last_fullsized":"2024-04-22T01:10:35.429532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175955+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:10:36.175955+0000","last_peered":"2024-04-22T01:10:36.175955+0000","last_clean":"2024-04-22T01:10:36.175955+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:10:36.175955+0000","last_undegraded":"2024-04-22T01:10:36.175955+0000","last_fullsized":"2024-04-22T01:10:36.175955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772526+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:10:35.772526+0000","last_peered":"2024-04-22T01:10:35.772526+0000","last_clean":"2024-04-22T01:10:35.772526+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:10:35.772526+0000","last_undegraded":"2024-04-22T01:10:35.772526+0000","last_fullsized":"2024-04-22T01:10:35.772526+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507195+0000","last_change":"2024-04-22T01:10:30.457611+0000","last_active":"2024-04-22T01:10:35.507195+0000","last_peered":"2024-04-22T01:10:35.507195+0000","last_clean":"2024-04-22T01:10:35.507195+0000","last_became_active":"2024-04-22T01:10:30.457272+0000","last_became_peered":"2024-04-22T01:10:30.457272+0000","last_unstale":"2024-04-22T01:10:35.507195+0000","last_undegraded":"2024-04-22T01:10:35.507195+0000","last_fullsized":"2024-04-22T01:10:35.507195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:11.233188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772028+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:10:35.772028+0000","last_peered":"2024-04-22T01:10:35.772028+0000","last_clean":"2024-04-22T01:10:35.772028+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:10:35.772028+0000","last_undegraded":"2024-04-22T01:10:35.772028+0000","last_fullsized":"2024-04-22T01:10:35.772028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429793+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:10:35.429793+0000","last_peered":"2024-04-22T01:10:35.429793+0000","last_clean":"2024-04-22T01:10:35.429793+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:10:35.429793+0000","last_undegraded":"2024-04-22T01:10:35.429793+0000","last_fullsized":"2024-04-22T01:10:35.429793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175476+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:10:36.175476+0000","last_peered":"2024-04-22T01:10:36.175476+0000","last_clean":"2024-04-22T01:10:36.175476+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:10:36.175476+0000","last_undegraded":"2024-04-22T01:10:36.175476+0000","last_fullsized":"2024-04-22T01:10:36.175476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:11:00.436259+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:11:00.436259+0000","last_peered":"2024-04-22T01:11:00.436259+0000","last_clean":"2024-04-22T01:11:00.436259+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:11:00.436259+0000","last_undegraded":"2024-04-22T01:11:00.436259+0000","last_fullsized":"2024-04-22T01:11:00.436259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429293+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:10:35.429293+0000","last_peered":"2024-04-22T01:10:35.429293+0000","last_clean":"2024-04-22T01:10:35.429293+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:10:35.429293+0000","last_undegraded":"2024-04-22T01:10:35.429293+0000","last_fullsized":"2024-04-22T01:10:35.429293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507688+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:10:35.507688+0000","last_peered":"2024-04-22T01:10:35.507688+0000","last_clean":"2024-04-22T01:10:35.507688+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:10:35.507688+0000","last_undegraded":"2024-04-22T01:10:35.507688+0000","last_fullsized":"2024-04-22T01:10:35.507688+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.771961+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:10:35.771961+0000","last_peered":"2024-04-22T01:10:35.771961+0000","last_clean":"2024-04-22T01:10:35.771961+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:10:35.771961+0000","last_undegraded":"2024-04-22T01:10:35.771961+0000","last_fullsized":"2024-04-22T01:10:35.771961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429207+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:10:35.429207+0000","last_peered":"2024-04-22T01:10:35.429207+0000","last_clean":"2024-04-22T01:10:35.429207+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:10:35.429207+0000","last_undegraded":"2024-04-22T01:10:35.429207+0000","last_fullsized":"2024-04-22T01:10:35.429207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175929+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:10:36.175929+0000","last_peered":"2024-04-22T01:10:36.175929+0000","last_clean":"2024-04-22T01:10:36.175929+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:10:36.175929+0000","last_undegraded":"2024-04-22T01:10:36.175929+0000","last_fullsized":"2024-04-22T01:10:36.175929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176345+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:10:36.176345+0000","last_peered":"2024-04-22T01:10:36.176345+0000","last_clean":"2024-04-22T01:10:36.176345+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:10:36.176345+0000","last_undegraded":"2024-04-22T01:10:36.176345+0000","last_fullsized":"2024-04-22T01:10:36.176345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772681+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:10:35.772681+0000","last_peered":"2024-04-22T01:10:35.772681+0000","last_clean":"2024-04-22T01:10:35.772681+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:10:35.772681+0000","last_undegraded":"2024-04-22T01:10:35.772681+0000","last_fullsized":"2024-04-22T01:10:35.772681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175826+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:10:36.175826+0000","last_peered":"2024-04-22T01:10:36.175826+0000","last_clean":"2024-04-22T01:10:36.175826+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:10:36.175826+0000","last_undegraded":"2024-04-22T01:10:36.175826+0000","last_fullsized":"2024-04-22T01:10:36.175826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.915086+0000","last_change":"2024-04-22T01:10:29.598225+0000","last_active":"2024-04-22T01:10:58.915086+0000","last_peered":"2024-04-22T01:10:58.915086+0000","last_clean":"2024-04-22T01:10:58.915086+0000","last_became_active":"2024-04-22T01:10:29.597867+0000","last_became_peered":"2024-04-22T01:10:29.597867+0000","last_unstale":"2024-04-22T01:10:58.915086+0000","last_undegraded":"2024-04-22T01:10:58.915086+0000","last_fullsized":"2024-04-22T01:10:58.915086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:52:32.759350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176125+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:10:36.176125+0000","last_peered":"2024-04-22T01:10:36.176125+0000","last_clean":"2024-04-22T01:10:36.176125+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:10:36.176125+0000","last_undegraded":"2024-04-22T01:10:36.176125+0000","last_fullsized":"2024-04-22T01:10:36.176125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429021+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:10:35.429021+0000","last_peered":"2024-04-22T01:10:35.429021+0000","last_clean":"2024-04-22T01:10:35.429021+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:10:35.429021+0000","last_undegraded":"2024-04-22T01:10:35.429021+0000","last_fullsized":"2024-04-22T01:10:35.429021+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:59.926196+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:10:59.926196+0000","last_peered":"2024-04-22T01:10:59.926196+0000","last_clean":"2024-04-22T01:10:59.926196+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:10:59.926196+0000","last_undegraded":"2024-04-22T01:10:59.926196+0000","last_fullsized":"2024-04-22T01:10:59.926196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175625+0000","last_change":"2024-04-22T01:10:30.472682+0000","last_active":"2024-04-22T01:10:36.175625+0000","last_peered":"2024-04-22T01:10:36.175625+0000","last_clean":"2024-04-22T01:10:36.175625+0000","last_became_active":"2024-04-22T01:10:30.472348+0000","last_became_peered":"2024-04-22T01:10:30.472348+0000","last_unstale":"2024-04-22T01:10:36.175625+0000","last_undegraded":"2024-04-22T01:10:36.175625+0000","last_fullsized":"2024-04-22T01:10:36.175625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:40:18.194268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967054+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:10:35.967054+0000","last_peered":"2024-04-22T01:10:35.967054+0000","last_clean":"2024-04-22T01:10:35.967054+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:10:35.967054+0000","last_undegraded":"2024-04-22T01:10:35.967054+0000","last_fullsized":"2024-04-22T01:10:35.967054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.891733+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:10:56.891733+0000","last_peered":"2024-04-22T01:10:56.891733+0000","last_clean":"2024-04-22T01:10:56.891733+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:10:56.891733+0000","last_undegraded":"2024-04-22T01:10:56.891733+0000","last_fullsized":"2024-04-22T01:10:56.891733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175661+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:10:36.175661+0000","last_peered":"2024-04-22T01:10:36.175661+0000","last_clean":"2024-04-22T01:10:36.175661+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:10:36.175661+0000","last_undegraded":"2024-04-22T01:10:36.175661+0000","last_fullsized":"2024-04-22T01:10:36.175661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.431360+0000","last_change":"2024-04-22T01:10:34.673139+0000","last_active":"2024-04-22T01:10:35.431360+0000","last_peered":"2024-04-22T01:10:35.431360+0000","last_clean":"2024-04-22T01:10:35.431360+0000","last_became_active":"2024-04-22T01:10:34.672004+0000","last_became_peered":"2024-04-22T01:10:34.672004+0000","last_unstale":"2024-04-22T01:10:35.431360+0000","last_undegraded":"2024-04-22T01:10:35.431360+0000","last_fullsized":"2024-04-22T01:10:35.431360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:08.609660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:59.925258+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:10:59.925258+0000","last_peered":"2024-04-22T01:10:59.925258+0000","last_clean":"2024-04-22T01:10:59.925258+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:10:59.925258+0000","last_undegraded":"2024-04-22T01:10:59.925258+0000","last_fullsized":"2024-04-22T01:10:59.925258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175718+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:10:36.175718+0000","last_peered":"2024-04-22T01:10:36.175718+0000","last_clean":"2024-04-22T01:10:36.175718+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:10:36.175718+0000","last_undegraded":"2024-04-22T01:10:36.175718+0000","last_fullsized":"2024-04-22T01:10:36.175718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.415911+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:10:36.415911+0000","last_peered":"2024-04-22T01:10:36.415911+0000","last_clean":"2024-04-22T01:10:36.415911+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:10:36.415911+0000","last_undegraded":"2024-04-22T01:10:36.415911+0000","last_fullsized":"2024-04-22T01:10:36.415911+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.884917+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:10:58.884917+0000","last_peered":"2024-04-22T01:10:58.884917+0000","last_clean":"2024-04-22T01:10:58.884917+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:10:58.884917+0000","last_undegraded":"2024-04-22T01:10:58.884917+0000","last_fullsized":"2024-04-22T01:10:58.884917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175894+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:10:36.175894+0000","last_peered":"2024-04-22T01:10:36.175894+0000","last_clean":"2024-04-22T01:10:36.175894+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:10:36.175894+0000","last_undegraded":"2024-04-22T01:10:36.175894+0000","last_fullsized":"2024-04-22T01:10:36.175894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.967286+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:10:35.967286+0000","last_peered":"2024-04-22T01:10:35.967286+0000","last_clean":"2024-04-22T01:10:35.967286+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:10:35.967286+0000","last_undegraded":"2024-04-22T01:10:35.967286+0000","last_fullsized":"2024-04-22T01:10:35.967286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:56.888623+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:10:56.888623+0000","last_peered":"2024-04-22T01:10:56.888623+0000","last_clean":"2024-04-22T01:10:56.888623+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:10:56.888623+0000","last_undegraded":"2024-04-22T01:10:56.888623+0000","last_fullsized":"2024-04-22T01:10:56.888623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429645+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:10:35.429645+0000","last_peered":"2024-04-22T01:10:35.429645+0000","last_clean":"2024-04-22T01:10:35.429645+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:10:35.429645+0000","last_undegraded":"2024-04-22T01:10:35.429645+0000","last_fullsized":"2024-04-22T01:10:35.429645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175824+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:10:36.175824+0000","last_peered":"2024-04-22T01:10:36.175824+0000","last_clean":"2024-04-22T01:10:36.175824+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:10:36.175824+0000","last_undegraded":"2024-04-22T01:10:36.175824+0000","last_fullsized":"2024-04-22T01:10:36.175824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:58.911974+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:10:58.911974+0000","last_peered":"2024-04-22T01:10:58.911974+0000","last_clean":"2024-04-22T01:10:58.911974+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:10:58.911974+0000","last_undegraded":"2024-04-22T01:10:58.911974+0000","last_fullsized":"2024-04-22T01:10:58.911974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429894+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:10:35.429894+0000","last_peered":"2024-04-22T01:10:35.429894+0000","last_clean":"2024-04-22T01:10:35.429894+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:10:35.429894+0000","last_undegraded":"2024-04-22T01:10:35.429894+0000","last_fullsized":"2024-04-22T01:10:35.429894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175085+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:10:36.175085+0000","last_peered":"2024-04-22T01:10:36.175085+0000","last_clean":"2024-04-22T01:10:36.175085+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:10:36.175085+0000","last_undegraded":"2024-04-22T01:10:36.175085+0000","last_fullsized":"2024-04-22T01:10:36.175085+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507825+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:10:35.507825+0000","last_peered":"2024-04-22T01:10:35.507825+0000","last_clean":"2024-04-22T01:10:35.507825+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:10:35.507825+0000","last_undegraded":"2024-04-22T01:10:35.507825+0000","last_fullsized":"2024-04-22T01:10:35.507825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429386+0000","last_change":"2024-04-22T01:10:30.471548+0000","last_active":"2024-04-22T01:10:35.429386+0000","last_peered":"2024-04-22T01:10:35.429386+0000","last_clean":"2024-04-22T01:10:35.429386+0000","last_became_active":"2024-04-22T01:10:30.471219+0000","last_became_peered":"2024-04-22T01:10:30.471219+0000","last_unstale":"2024-04-22T01:10:35.429386+0000","last_undegraded":"2024-04-22T01:10:35.429386+0000","last_fullsized":"2024-04-22T01:10:35.429386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:27.776929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507030+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:10:35.507030+0000","last_peered":"2024-04-22T01:10:35.507030+0000","last_clean":"2024-04-22T01:10:35.507030+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:10:35.507030+0000","last_undegraded":"2024-04-22T01:10:35.507030+0000","last_fullsized":"2024-04-22T01:10:35.507030+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772488+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:10:35.772488+0000","last_peered":"2024-04-22T01:10:35.772488+0000","last_clean":"2024-04-22T01:10:35.772488+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:10:35.772488+0000","last_undegraded":"2024-04-22T01:10:35.772488+0000","last_fullsized":"2024-04-22T01:10:35.772488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429911+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:10:35.429911+0000","last_peered":"2024-04-22T01:10:35.429911+0000","last_clean":"2024-04-22T01:10:35.429911+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:10:35.429911+0000","last_undegraded":"2024-04-22T01:10:35.429911+0000","last_fullsized":"2024-04-22T01:10:35.429911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176034+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:10:36.176034+0000","last_peered":"2024-04-22T01:10:36.176034+0000","last_clean":"2024-04-22T01:10:36.176034+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:10:36.176034+0000","last_undegraded":"2024-04-22T01:10:36.176034+0000","last_fullsized":"2024-04-22T01:10:36.176034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429125+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:10:35.429125+0000","last_peered":"2024-04-22T01:10:35.429125+0000","last_clean":"2024-04-22T01:10:35.429125+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:10:35.429125+0000","last_undegraded":"2024-04-22T01:10:35.429125+0000","last_fullsized":"2024-04-22T01:10:35.429125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429016+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:10:35.429016+0000","last_peered":"2024-04-22T01:10:35.429016+0000","last_clean":"2024-04-22T01:10:35.429016+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:10:35.429016+0000","last_undegraded":"2024-04-22T01:10:35.429016+0000","last_fullsized":"2024-04-22T01:10:35.429016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.175214+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:10:36.175214+0000","last_peered":"2024-04-22T01:10:36.175214+0000","last_clean":"2024-04-22T01:10:36.175214+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:10:36.175214+0000","last_undegraded":"2024-04-22T01:10:36.175214+0000","last_fullsized":"2024-04-22T01:10:36.175214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772601+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:10:35.772601+0000","last_peered":"2024-04-22T01:10:35.772601+0000","last_clean":"2024-04-22T01:10:35.772601+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:10:35.772601+0000","last_undegraded":"2024-04-22T01:10:35.772601+0000","last_fullsized":"2024-04-22T01:10:35.772601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176661+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:10:36.176661+0000","last_peered":"2024-04-22T01:10:36.176661+0000","last_clean":"2024-04-22T01:10:36.176661+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:10:36.176661+0000","last_undegraded":"2024-04-22T01:10:36.176661+0000","last_fullsized":"2024-04-22T01:10:36.176661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.429670+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:10:35.429670+0000","last_peered":"2024-04-22T01:10:35.429670+0000","last_clean":"2024-04-22T01:10:35.429670+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:10:35.429670+0000","last_undegraded":"2024-04-22T01:10:35.429670+0000","last_fullsized":"2024-04-22T01:10:35.429670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176275+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:10:36.176275+0000","last_peered":"2024-04-22T01:10:36.176275+0000","last_clean":"2024-04-22T01:10:36.176275+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:10:36.176275+0000","last_undegraded":"2024-04-22T01:10:36.176275+0000","last_fullsized":"2024-04-22T01:10:36.176275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.507188+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:10:35.507188+0000","last_peered":"2024-04-22T01:10:35.507188+0000","last_clean":"2024-04-22T01:10:35.507188+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:10:35.507188+0000","last_undegraded":"2024-04-22T01:10:35.507188+0000","last_fullsized":"2024-04-22T01:10:35.507188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176375+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:10:36.176375+0000","last_peered":"2024-04-22T01:10:36.176375+0000","last_clean":"2024-04-22T01:10:36.176375+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:10:36.176375+0000","last_undegraded":"2024-04-22T01:10:36.176375+0000","last_fullsized":"2024-04-22T01:10:36.176375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176199+0000","last_change":"2024-04-22T01:10:30.477852+0000","last_active":"2024-04-22T01:10:36.176199+0000","last_peered":"2024-04-22T01:10:36.176199+0000","last_clean":"2024-04-22T01:10:36.176199+0000","last_became_active":"2024-04-22T01:10:30.477606+0000","last_became_peered":"2024-04-22T01:10:30.477606+0000","last_unstale":"2024-04-22T01:10:36.176199+0000","last_undegraded":"2024-04-22T01:10:36.176199+0000","last_fullsized":"2024-04-22T01:10:36.176199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:21:39.983985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:35.772305+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:10:35.772305+0000","last_peered":"2024-04-22T01:10:35.772305+0000","last_clean":"2024-04-22T01:10:35.772305+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:10:35.772305+0000","last_undegraded":"2024-04-22T01:10:35.772305+0000","last_fullsized":"2024-04-22T01:10:35.772305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T01:10:36.176240+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:10:36.176240+0000","last_peered":"2024-04-22T01:10:36.176240+0000","last_clean":"2024-04-22T01:10:36.176240+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:10:36.176240+0000","last_undegraded":"2024-04-22T01:10:36.176240+0000","last_fullsized":"2024-04-22T01:10:36.176240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":51112,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":23,"num_read_kb":63,"num_write":176,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":73520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":478407,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1109790,"data_compressed":51092,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":482433,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":51092,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":489409,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477623,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920824,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477915,"data_compressed":14679,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":51092,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":12194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":47407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:11:02.749 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T01:11:02.750 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T01:11:02.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T01:11:03.033 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:03.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:02 smithi038 ceph-mon[28043]: pgmap v312: 129 pgs: 129 active+clean; 627 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-04-22T01:11:03.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:02 smithi038 ceph-mon[28043]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T01:11:03.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:02 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/44614490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:03.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:02 smithi038 ceph-mon[28043]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:11:03.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:02 smithi203 ceph-mon[31825]: pgmap v312: 129 pgs: 129 active+clean; 627 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-04-22T01:11:03.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:02 smithi203 ceph-mon[31825]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T01:11:03.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:02 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/44614490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:03.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:02 smithi203 ceph-mon[31825]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:11:03.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:02 smithi111 ceph-mon[31057]: pgmap v312: 129 pgs: 129 active+clean; 627 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-04-22T01:11:03.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:02 smithi111 ceph-mon[31057]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T01:11:03.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:02 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/44614490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:03.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:02 smithi111 ceph-mon[31057]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:11:04.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:03 smithi203 ceph-mon[31825]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:11:04.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:03 smithi038 ceph-mon[28043]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:11:04.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:03 smithi111 ceph-mon[31057]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 -- 172.21.15.38:0/3882012643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e8103e60 msgr2=0x7f91e810a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 --2- 172.21.15.38:0/3882012643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e8103e60 0x7f91e810a710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f91e000b0d0 tx=0x7f91e000b3e0 comp rx=0 tx=0).stop 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 -- 172.21.15.38:0/3882012643 shutdown_connections 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 --2- 172.21.15.38:0/3882012643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e8103e60 0x7f91e810a710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 --2- 172.21.15.38:0/3882012643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91e81022c0 0x7f91e81026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 --2- 172.21.15.38:0/3882012643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e81034c0 0x7f91e8103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 -- 172.21.15.38:0/3882012643 >> 172.21.15.38:0/3882012643 conn(0x7f91e80fd880 msgr2=0x7f91e80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 -- 172.21.15.38:0/3882012643 shutdown_connections 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 -- 172.21.15.38:0/3882012643 wait complete. 2024-04-22T01:11:04.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.917+0000 7f91f04c0700 1 Processor -- start 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 -- start start 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91e81022c0 0x7f91e81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 0x7f91e8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e8103e60 0x7f91e819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91e810dfa0 con 0x7f91e81034c0 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91e810de20 con 0x7f91e8103e60 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91f04c0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91e810e120 con 0x7f91e81022c0 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 0x7f91e8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 0x7f91e8198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44528/0 (socket says 172.21.15.38:44528) 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 -- 172.21.15.38:0/177277234 learned_addr learned my addr 172.21.15.38:0/177277234 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91ee25c700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91e81022c0 0x7f91e81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eea5d700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e8103e60 0x7f91e819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 -- 172.21.15.38:0/177277234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91e81022c0 msgr2=0x7f91e81984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91e81022c0 0x7f91e81984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:04.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 -- 172.21.15.38:0/177277234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e8103e60 msgr2=0x7f91e819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:04.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e8103e60 0x7f91e819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:04.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.918+0000 7f91eda5b700 1 -- 172.21.15.38:0/177277234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e81a0150 con 0x7f91e81034c0 2024-04-22T01:11:04.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.919+0000 7f91eda5b700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 0x7f91e8198a30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b560 tx=0x7f91d800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.919+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8015020 con 0x7f91e81034c0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.919+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e81a0440 con 0x7f91e81034c0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.919+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e81a0a50 con 0x7f91e81034c0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.919+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d8011d10 con 0x7f91e81034c0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.919+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d800f2e0 con 0x7f91e81034c0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.920+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91cc005360 con 0x7f91e81034c0 2024-04-22T01:11:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.921+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f91d8018020 con 0x7f91e81034c0 2024-04-22T01:11:04.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.922+0000 7f91df7fe700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d40a2d60 0x7f91d40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:04.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.922+0000 7f91ee25c700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d40a2d60 0x7f91d40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:04.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.922+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f91d80cedf0 con 0x7f91e81034c0 2024-04-22T01:11:04.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.923+0000 7f91ee25c700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d40a2d60 0x7f91d40a5220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f91e8103320 tx=0x7f91e4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:04.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:04.927+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f91d801f020 con 0x7f91e81034c0 2024-04-22T01:11:05.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.220+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f91cc005650 con 0x7f91e81034c0 2024-04-22T01:11:05.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:04 smithi038 ceph-mon[28043]: pgmap v313: 129 pgs: 129 active+clean; 628 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 8.2 KiB/s wr, 17 op/s 2024-04-22T01:11:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.223+0000 7f91df7fe700 1 -- 172.21.15.38:0/177277234 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f91d809bac0 con 0x7f91e81034c0 2024-04-22T01:11:05.223 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-04-22T01:11:05.223 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQApuSVmx+VEDRAASHi/EgM4CXw5dT/6tf4qrQ== 2024-04-22T01:11:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d40a2d60 msgr2=0x7f91d40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d40a2d60 0x7f91d40a5220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f91e8103320 tx=0x7f91e4009190 comp rx=0 tx=0).stop 2024-04-22T01:11:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 msgr2=0x7f91e8198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 0x7f91e8198a30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b560 tx=0x7f91d800b870 comp rx=0 tx=0).stop 2024-04-22T01:11:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 shutdown_connections 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91d40a2d60 0x7f91d40a5220 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e8103e60 0x7f91e819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91e81022c0 0x7f91e81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 --2- 172.21.15.38:0/177277234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91e81034c0 0x7f91e8198a30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 >> 172.21.15.38:0/177277234 conn(0x7f91e80fd880 msgr2=0x7f91e81066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 shutdown_connections 2024-04-22T01:11:05.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:05.225+0000 7f91f04c0700 1 -- 172.21.15.38:0/177277234 wait complete. 2024-04-22T01:11:05.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:04 smithi203 ceph-mon[31825]: pgmap v313: 129 pgs: 129 active+clean; 628 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 8.2 KiB/s wr, 17 op/s 2024-04-22T01:11:05.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:04 smithi111 ceph-mon[31057]: pgmap v313: 129 pgs: 129 active+clean; 628 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 8.2 KiB/s wr, 17 op/s 2024-04-22T01:11:06.009 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:11:06.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T01:11:06.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T01:11:06.047 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T01:11:06.047 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T01:11:06.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T01:11:06.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:05 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/177277234' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T01:11:06.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:05 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/177277234' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T01:11:06.343 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:06.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:05 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/177277234' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T01:11:06.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:05 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/177277234' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T01:11:06.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/177277234' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T01:11:06.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/177277234' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T01:11:07.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:07 smithi038 ceph-mon[28043]: pgmap v314: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2024-04-22T01:11:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:07 smithi203 ceph-mon[31825]: pgmap v314: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2024-04-22T01:11:07.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:07 smithi111 ceph-mon[31057]: pgmap v314: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.309+0000 7f160335f700 1 -- 172.21.15.38:0/995071315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc107570 msgr2=0x7f15fc10f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.309+0000 7f160335f700 1 --2- 172.21.15.38:0/995071315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc107570 0x7f15fc10f140 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f15f400b0a0 tx=0x7f15f400b3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 -- 172.21.15.38:0/995071315 shutdown_connections 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 --2- 172.21.15.38:0/995071315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc107570 0x7f15fc10f140 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 --2- 172.21.15.38:0/995071315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15fc1024e0 0x7f15fc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 --2- 172.21.15.38:0/995071315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc102e90 0x7f15fc106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 -- 172.21.15.38:0/995071315 >> 172.21.15.38:0/995071315 conn(0x7f15fc079390 msgr2=0x7f15fc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 -- 172.21.15.38:0/995071315 shutdown_connections 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 -- 172.21.15.38:0/995071315 wait complete. 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 Processor -- start 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.310+0000 7f160335f700 1 -- start start 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f160335f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 0x7f15fc074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f160335f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc102e90 0x7f15fc074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f160335f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15fc107570 0x7f15fc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f160335f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc112550 con 0x7f15fc1024e0 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f160335f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc1123d0 con 0x7f15fc107570 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f160335f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc1126d0 con 0x7f15fc102e90 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 0x7f15fc074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 0x7f15fc074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47164/0 (socket says 172.21.15.38:47164) 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 -- 172.21.15.38:0/2193777337 learned_addr learned my addr 172.21.15.38:0/2193777337 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16008fa700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc102e90 0x7f15fc074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16018fc700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15fc107570 0x7f15fc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 -- 172.21.15.38:0/2193777337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc102e90 msgr2=0x7f15fc074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc102e90 0x7f15fc074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 -- 172.21.15.38:0/2193777337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15fc107570 msgr2=0x7f15fc074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15fc107570 0x7f15fc074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 -- 172.21.15.38:0/2193777337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc1a6e70 con 0x7f15fc1024e0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f16010fb700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 0x7f15fc074530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f15f800d950 tx=0x7f15f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.311+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8016070 con 0x7f15fc1024e0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.312+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15fc1a7160 con 0x7f15fc1024e0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.312+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f800fd70 con 0x7f15fc1024e0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.312+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f800e610 con 0x7f15fc1024e0 2024-04-22T01:11:08.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.312+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15fc1a76a0 con 0x7f15fc1024e0 2024-04-22T01:11:08.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.312+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15fc068f20 con 0x7f15fc1024e0 2024-04-22T01:11:08.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.314+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f15f8019070 con 0x7f15fc1024e0 2024-04-22T01:11:08.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.314+0000 7f15f27fc700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15e80a2cb0 0x7f15e80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:08.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.314+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f15f80cee80 con 0x7f15fc1024e0 2024-04-22T01:11:08.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.314+0000 7f16008fa700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15e80a2cb0 0x7f15e80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:08.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.315+0000 7f16008fa700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15e80a2cb0 0x7f15e80a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15fc108870 tx=0x7f15ec00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.318+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f15f809ba40 con 0x7f15fc1024e0 2024-04-22T01:11:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.586+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f15fc075a60 con 0x7f15fc1024e0 2024-04-22T01:11:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.591+0000 7f15f27fc700 1 -- 172.21.15.38:0/2193777337 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263198 (secure 0 0 0) 0x7f15f809b340 con 0x7f15fc1024e0 2024-04-22T01:11:08.595 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15e80a2cb0 msgr2=0x7f15e80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15e80a2cb0 0x7f15e80a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15fc108870 tx=0x7f15ec00b3c0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 msgr2=0x7f15fc074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 0x7f15fc074530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f15f800d950 tx=0x7f15f800dc60 comp rx=0 tx=0).stop 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 shutdown_connections 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15e80a2cb0 0x7f15e80a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15fc107570 0x7f15fc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15fc1024e0 0x7f15fc074530 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 --2- 172.21.15.38:0/2193777337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc102e90 0x7f15fc074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 >> 172.21.15.38:0/2193777337 conn(0x7f15fc079390 msgr2=0x7f15fc100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 shutdown_connections 2024-04-22T01:11:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:08.594+0000 7f160335f700 1 -- 172.21.15.38:0/2193777337 wait complete. 2024-04-22T01:11:09.377 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":3086698001},{"type":"v1","addr":"172.21.15.38:6801","nonce":3086698001}]},"active_addr":"172.21.15.38:6801/3086698001","active_change":"2024-04-22T01:03:15.696083+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":486040671}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":84523250}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2298360886}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1392401979}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2173975479}]}]} 2024-04-22T01:11:09.382 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T01:11:09.382 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T01:11:09.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T01:11:09.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:09 smithi203 ceph-mon[31825]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-04-22T01:11:09.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:09 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2193777337' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T01:11:09.640 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:09 smithi111 ceph-mon[31057]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-04-22T01:11:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:09 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2193777337' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T01:11:09.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:09 smithi038 ceph-mon[28043]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-04-22T01:11:09.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:09 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2193777337' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T01:11:11.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 -- 172.21.15.38:0/3401384513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 msgr2=0x7f83e410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 --2- 172.21.15.38:0/3401384513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e410f160 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0099c0 tx=0x7f83dc009cd0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 -- 172.21.15.38:0/3401384513 shutdown_connections 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 --2- 172.21.15.38:0/3401384513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e410f160 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 --2- 172.21.15.38:0/3401384513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e406ac40 0x7f83e406b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 --2- 172.21.15.38:0/3401384513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e406b5f0 0x7f83e4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 -- 172.21.15.38:0/3401384513 >> 172.21.15.38:0/3401384513 conn(0x7f83e40781d0 msgr2=0x7f83e40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 -- 172.21.15.38:0/3401384513 shutdown_connections 2024-04-22T01:11:11.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.482+0000 7f83e8e34700 1 -- 172.21.15.38:0/3401384513 wait complete. 2024-04-22T01:11:11.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:11 smithi038 ceph-mon[28043]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-04-22T01:11:11.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.483+0000 7f83e8e34700 1 Processor -- start 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.483+0000 7f83e8e34700 1 -- start start 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e8e34700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e406ac40 0x7f83e41011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e8e34700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e406b5f0 0x7f83e41016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e8e34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e4101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e8e34700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f83e4112590 con 0x7f83e4107590 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e8e34700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f83e4112410 con 0x7f83e406b5f0 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e8e34700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83e4112710 con 0x7f83e406ac40 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e4101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e406ac40 0x7f83e41011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e4101c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47196/0 (socket says 172.21.15.38:47196) 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 -- 172.21.15.38:0/758828742 learned_addr learned my addr 172.21.15.38:0/758828742 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e1d9b700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e406b5f0 0x7f83e41016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:11.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 -- 172.21.15.38:0/758828742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e406ac40 msgr2=0x7f83e41011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e406ac40 0x7f83e41011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 -- 172.21.15.38:0/758828742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e406b5f0 msgr2=0x7f83e41016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e406b5f0 0x7f83e41016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 -- 172.21.15.38:0/758828742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e41a6f30 con 0x7f83e4107590 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.484+0000 7f83e2d9d700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e4101c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0059d0 tx=0x7f83dc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.485+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc027070 con 0x7f83e4107590 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.485+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e41a71c0 con 0x7f83e4107590 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.485+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83dc003740 con 0x7f83e4107590 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.485+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc022bf0 con 0x7f83e4107590 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.485+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e41a7610 con 0x7f83e4107590 2024-04-22T01:11:11.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.486+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e4068f20 con 0x7f83e4107590 2024-04-22T01:11:11.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.487+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f83dc02a070 con 0x7f83e4107590 2024-04-22T01:11:11.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.488+0000 7f83d37fe700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83cc0a2cb0 0x7f83cc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:11.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.488+0000 7f83e259c700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83cc0a2cb0 0x7f83cc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:11.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.488+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f83dc0d08d0 con 0x7f83e4107590 2024-04-22T01:11:11.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.489+0000 7f83e259c700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83cc0a2cb0 0x7f83cc0a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f83d40060b0 tx=0x7f83d400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:11.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.492+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f83dc031070 con 0x7f83e4107590 2024-04-22T01:11:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:11 smithi203 ceph-mon[31825]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-04-22T01:11:11.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:11 smithi111 ceph-mon[31057]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-04-22T01:11:11.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.703+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83e402dd90 con 0x7f83e4107590 2024-04-22T01:11:11.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.704+0000 7f83d37fe700 1 -- 172.21.15.38:0/758828742 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f83dc013070 con 0x7f83e4107590 2024-04-22T01:11:11.705 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:11.705 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83cc0a2cb0 msgr2=0x7f83cc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83cc0a2cb0 0x7f83cc0a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f83d40060b0 tx=0x7f83d400c040 comp rx=0 tx=0).stop 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 msgr2=0x7f83e4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e4101c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0059d0 tx=0x7f83dc009f80 comp rx=0 tx=0).stop 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 shutdown_connections 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f83cc0a2cb0 0x7f83cc0a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e4107590 0x7f83e4101c20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e406ac40 0x7f83e41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 --2- 172.21.15.38:0/758828742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e406b5f0 0x7f83e41016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 >> 172.21.15.38:0/758828742 conn(0x7f83e40781d0 msgr2=0x7f83e40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 shutdown_connections 2024-04-22T01:11:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:11.707+0000 7f83e8e34700 1 -- 172.21.15.38:0/758828742 wait complete. 2024-04-22T01:11:12.388 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T01:11:12.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T01:11:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:12.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:12 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/758828742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:12.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:12 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/758828742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:12.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:12 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/758828742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:13.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:13 smithi038 ceph-mon[28043]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 24 op/s 2024-04-22T01:11:13.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:13 smithi111 ceph-mon[31057]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 24 op/s 2024-04-22T01:11:13.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:13 smithi203 ceph-mon[31825]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 24 op/s 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 -- 172.21.15.38:0/1182047830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350107570 msgr2=0x7f435010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 --2- 172.21.15.38:0/1182047830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350107570 0x7f435010f140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f434800b0a0 tx=0x7f434800b3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 -- 172.21.15.38:0/1182047830 shutdown_connections 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 --2- 172.21.15.38:0/1182047830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350107570 0x7f435010f140 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 --2- 172.21.15.38:0/1182047830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43501024e0 0x7f43501028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 --2- 172.21.15.38:0/1182047830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4350102e90 0x7f4350106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 -- 172.21.15.38:0/1182047830 >> 172.21.15.38:0/1182047830 conn(0x7f4350079390 msgr2=0x7f43500797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 -- 172.21.15.38:0/1182047830 shutdown_connections 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 -- 172.21.15.38:0/1182047830 wait complete. 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.368+0000 7f4355da0700 1 Processor -- start 2024-04-22T01:11:14.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 -- start start 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43501024e0 0x7f435019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 0x7f435019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4350107570 0x7f4350198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4350112550 con 0x7f4350102e90 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f43501123d0 con 0x7f4350107570 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f4355da0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f43501126d0 con 0x7f43501024e0 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f434effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 0x7f435019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f434effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 0x7f435019eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47208/0 (socket says 172.21.15.38:47208) 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f434effd700 1 -- 172.21.15.38:0/1988310495 learned_addr learned my addr 172.21.15.38:0/1988310495 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f434ffff700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4350107570 0x7f4350198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.369+0000 7f434f7fe700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43501024e0 0x7f435019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434effd700 1 -- 172.21.15.38:0/1988310495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43501024e0 msgr2=0x7f435019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:14.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434effd700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43501024e0 0x7f435019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434effd700 1 -- 172.21.15.38:0/1988310495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4350107570 msgr2=0x7f4350198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434effd700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4350107570 0x7f4350198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434effd700 1 -- 172.21.15.38:0/1988310495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4350198ff0 con 0x7f4350102e90 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434effd700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 0x7f435019eb80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f433800c9e0 tx=0x7f433800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4338007aa0 con 0x7f4350102e90 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4338007c40 con 0x7f4350102e90 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4338018660 con 0x7f4350102e90 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4350199280 con 0x7f4350102e90 2024-04-22T01:11:14.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.370+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4350074630 con 0x7f4350102e90 2024-04-22T01:11:14.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.371+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43501995f0 con 0x7f4350102e90 2024-04-22T01:11:14.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.372+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f433800f450 con 0x7f4350102e90 2024-04-22T01:11:14.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.373+0000 7f434cff9700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f433c0a2cb0 0x7f433c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:14.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.373+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f43380cf330 con 0x7f4350102e90 2024-04-22T01:11:14.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.373+0000 7f434f7fe700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f433c0a2cb0 0x7f433c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:14.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.373+0000 7f434f7fe700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f433c0a2cb0 0x7f433c0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4340005950 tx=0x7f434000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:14.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.377+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f43501995f0 con 0x7f4350102e90 2024-04-22T01:11:14.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.584+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4350068f20 con 0x7f4350102e90 2024-04-22T01:11:14.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.586+0000 7f434cff9700 1 -- 172.21.15.38:0/1988310495 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f4338014030 con 0x7f4350102e90 2024-04-22T01:11:14.586 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:11:14.586 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:10:35.016576+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:11:14.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f433c0a2cb0 msgr2=0x7f433c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f433c0a2cb0 0x7f433c0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4340005950 tx=0x7f434000b410 comp rx=0 tx=0).stop 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 msgr2=0x7f435019eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 0x7f435019eb80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f433800c9e0 tx=0x7f433800ccf0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 shutdown_connections 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f433c0a2cb0 0x7f433c0a5170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4350107570 0x7f4350198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43501024e0 0x7f435019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 --2- 172.21.15.38:0/1988310495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4350102e90 0x7f435019eb80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 >> 172.21.15.38:0/1988310495 conn(0x7f4350079390 msgr2=0x7f4350100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 shutdown_connections 2024-04-22T01:11:14.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:14.588+0000 7f4355da0700 1 -- 172.21.15.38:0/1988310495 wait complete. 2024-04-22T01:11:15.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T01:11:15.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T01:11:15.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T01:11:15.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T01:11:15.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T01:11:15.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T01:11:15.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T01:11:15.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T01:11:15.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-22T01:11:15.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-22T01:11:15.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-22T01:11:15.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-22T01:11:15.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:15 smithi038 ceph-mon[28043]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 KiB/s wr, 15 op/s 2024-04-22T01:11:15.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:15 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1988310495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:15.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:15 smithi111 ceph-mon[31057]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 KiB/s wr, 15 op/s 2024-04-22T01:11:15.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:15 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1988310495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:15 smithi203 ceph-mon[31825]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 KiB/s wr, 15 op/s 2024-04-22T01:11:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:15 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1988310495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:11:15.989 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:15.993 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:15.998 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:15.999 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:16.001 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:16.005 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:16.066 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:16.102 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:16.102 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-22T01:11:16.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T01:11:16.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-04-22T01:11:16.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-22T01:11:16.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-22T01:11:16.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-22T01:11:16.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-04-22T01:11:16.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-22T01:11:16.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-04-22T01:11:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-22T01:11:16.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-22T01:11:17.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T01:11:17.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-04-22T01:11:17.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-22T01:11:17.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-22T01:11:17.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:17 smithi038 ceph-mon[28043]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T01:11:17.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-22T01:11:17.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-22T01:11:17.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-04-22T01:11:17.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-22T01:11:17.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-22T01:11:17.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:17 smithi111 ceph-mon[31057]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-22T01:11:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T01:11:17.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-04-22T01:11:17.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-22T01:11:17.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-22T01:11:17.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:17 smithi203 ceph-mon[31825]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T01:11:17.842 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:17.842 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:17.842 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:18.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:18 smithi111 ceph-mon[31057]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T01:11:18.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:18 smithi038 ceph-mon[28043]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T01:11:18.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:18 smithi203 ceph-mon[31825]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T01:11:19.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:19 smithi038 ceph-mon[28043]: pgmap v322: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 895 B/s wr, 12 op/s 2024-04-22T01:11:19.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:19 smithi111 ceph-mon[31057]: pgmap v322: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 895 B/s wr, 12 op/s 2024-04-22T01:11:19.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:19 smithi203 ceph-mon[31825]: pgmap v322: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 895 B/s wr, 12 op/s 2024-04-22T01:11:21.618 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:21 smithi038 ceph-mon[28043]: pgmap v323: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T01:11:21.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:21 smithi111 ceph-mon[31057]: pgmap v323: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T01:11:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:21 smithi203 ceph-mon[31825]: pgmap v323: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T01:11:23.382 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:23 smithi038 ceph-mon[28043]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:11:23.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:23 smithi111 ceph-mon[31057]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:11:23.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:23 smithi203 ceph-mon[31825]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:11:25.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:25 smithi038 ceph-mon[28043]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:11:25.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:25 smithi111 ceph-mon[31057]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:11:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:25 smithi203 ceph-mon[31825]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.366+0000 7fe73ad8e700 1 -- 172.21.15.38:0/2900036835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734107590 msgr2=0x7fe73410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.366+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/2900036835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734107590 0x7fe73410f160 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00b0a0 tx=0x7fe72c00b3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 -- 172.21.15.38:0/2900036835 shutdown_connections 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/2900036835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734107590 0x7fe73410f160 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/2900036835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe734102510 0x7fe7341028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/2900036835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe734102ec0 0x7fe734106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 -- 172.21.15.38:0/2900036835 >> 172.21.15.38:0/2900036835 conn(0x7fe7340793a0 msgr2=0x7fe7340797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:26.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 -- 172.21.15.38:0/2900036835 shutdown_connections 2024-04-22T01:11:26.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 -- 172.21.15.38:0/2900036835 wait complete. 2024-04-22T01:11:26.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 Processor -- start 2024-04-22T01:11:26.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.367+0000 7fe73ad8e700 1 -- start start 2024-04-22T01:11:26.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73ad8e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734102510 0x7fe73410ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73ad8e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe734102ec0 0x7fe73410bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73ad8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe734107590 0x7fe73410c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734102510 0x7fe73410ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734102510 0x7fe73410ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39036/0 (socket says 172.21.15.38:39036) 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73ad8e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe734075c00 con 0x7fe734102510 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 -- 172.21.15.38:0/43756155 learned_addr learned my addr 172.21.15.38:0/43756155 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe734075a80 con 0x7fe734107590 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe734075d80 con 0x7fe734102ec0 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe73932b700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe734107590 0x7fe73410c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe733fff700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe734102ec0 0x7fe73410bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 -- 172.21.15.38:0/43756155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe734102ec0 msgr2=0x7fe73410bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe734102ec0 0x7fe73410bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 -- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe734107590 msgr2=0x7fe73410c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe734107590 0x7fe73410c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe738b2a700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73410c870 con 0x7fe734102510 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.368+0000 7fe733fff700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe734102ec0 0x7fe73410bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.369+0000 7fe738b2a700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734102510 0x7fe73410ec70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe72400e9a0 tx=0x7fe72400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.369+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72400cbd0 con 0x7fe734102510 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.369+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe72400cd70 con 0x7fe734102510 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.369+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73410cb60 con 0x7fe734102510 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.369+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe724018750 con 0x7fe734102510 2024-04-22T01:11:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.369+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7341aba50 con 0x7fe734102510 2024-04-22T01:11:26.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.370+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe734100b80 con 0x7fe734102510 2024-04-22T01:11:26.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.371+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe724010430 con 0x7fe734102510 2024-04-22T01:11:26.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe731ffb700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe71c0a2cb0 0x7fe71c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe733fff700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe71c0a2cb0 0x7fe71c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe7240cf7f0 con 0x7fe734102510 2024-04-22T01:11:26.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe731ffb700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] conn(0x7fe71c0ade10 0x7fe71c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe71c0b0980 con 0x7fe71c0ade10 2024-04-22T01:11:26.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe7240d42c0 con 0x7fe734102510 2024-04-22T01:11:26.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.372+0000 7fe733fff700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe71c0a2cb0 0x7fe71c0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe728005fd0 tx=0x7fe728005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:26.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.373+0000 7fe73932b700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] conn(0x7fe71c0ade10 0x7fe71c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.373+0000 7fe73932b700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] conn(0x7fe71c0ade10 0x7fe71c0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:26.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.374+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== osd.7 v2:172.21.15.111:6816/3257841658 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe71c0b0980 con 0x7fe71c0ade10 2024-04-22T01:11:26.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.391+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 --> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe734103990 con 0x7fe71c0ade10 2024-04-22T01:11:26.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.391+0000 7fe731ffb700 1 -- 172.21.15.38:0/43756155 <== osd.7 v2:172.21.15.111:6816/3257841658 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe734103990 con 0x7fe71c0ade10 2024-04-22T01:11:26.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] conn(0x7fe71c0ade10 msgr2=0x7fe71c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] conn(0x7fe71c0ade10 0x7fe71c0b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe71c0a2cb0 msgr2=0x7fe71c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe71c0a2cb0 0x7fe71c0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe728005fd0 tx=0x7fe728005d90 comp rx=0 tx=0).stop 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734102510 msgr2=0x7fe73410ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 --2- 172.21.15.38:0/43756155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe734102510 0x7fe73410ec70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe72400e9a0 tx=0x7fe72400ecb0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73932b700 1 -- 172.21.15.38:0/43756155 reap_dead start 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 shutdown_connections 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 >> 172.21.15.38:0/43756155 conn(0x7fe7340793a0 msgr2=0x7fe734100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 shutdown_connections 2024-04-22T01:11:26.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.392+0000 7fe73ad8e700 1 -- 172.21.15.38:0/43756155 wait complete. 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 -- 172.21.15.38:0/3546141866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac1082c0 msgr2=0x7fb2ac1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 --2- 172.21.15.38:0/3546141866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac1082c0 0x7fb2ac1086a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009990 tx=0x7fb2a0009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 -- 172.21.15.38:0/3546141866 shutdown_connections 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 --2- 172.21.15.38:0/3546141866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ac1011c0 0x7fb2ac10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 --2- 172.21.15.38:0/3546141866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac1082c0 0x7fb2ac1086a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 --2- 172.21.15.38:0/3546141866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2ac100820 0x7fb2ac100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 -- 172.21.15.38:0/3546141866 >> 172.21.15.38:0/3546141866 conn(0x7fb2ac0777a0 msgr2=0x7fb2ac0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 -- 172.21.15.38:0/3546141866 shutdown_connections 2024-04-22T01:11:26.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.969+0000 7fb2b1188700 1 -- 172.21.15.38:0/3546141866 wait complete. 2024-04-22T01:11:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 Processor -- start 2024-04-22T01:11:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 -- start start 2024-04-22T01:11:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac100820 0x7fb2ac19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ac1011c0 0x7fb2ac19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2ac1082c0 0x7fb2ac198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac112520 con 0x7fb2ac100820 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac1123a0 con 0x7fb2ac1082c0 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.970+0000 7fb2b1188700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac1126a0 con 0x7fb2ac1011c0 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac100820 0x7fb2ac19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac100820 0x7fb2ac19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34402/0 (socket says 172.21.15.38:34402) 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 -- 172.21.15.38:0/1973514043 learned_addr learned my addr 172.21.15.38:0/1973514043 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2ab59e700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2ac1082c0 0x7fb2ac198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aa59c700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ac1011c0 0x7fb2ac19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 -- 172.21.15.38:0/1973514043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ac1011c0 msgr2=0x7fb2ac19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2ac1011c0 0x7fb2ac19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 -- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2ac1082c0 msgr2=0x7fb2ac198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2ac1082c0 0x7fb2ac198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 -- 172.21.15.38:0/1973514043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac198fe0 con 0x7fb2ac100820 2024-04-22T01:11:26.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2aad9d700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac100820 0x7fb2ac19e680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0000c00 tx=0x7fb2a000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0027070 con 0x7fb2ac100820 2024-04-22T01:11:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2a0003d70 con 0x7fb2ac100820 2024-04-22T01:11:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac199270 con 0x7fb2ac100820 2024-04-22T01:11:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a000f6c0 con 0x7fb2ac100820 2024-04-22T01:11:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.971+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac1a54e0 con 0x7fb2ac100820 2024-04-22T01:11:26.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.972+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2ac0629c0 con 0x7fb2ac100820 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.973+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb2a002a070 con 0x7fb2ac100820 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.974+0000 7fb293fff700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2940a2cb0 0x7fb2940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.974+0000 7fb2aa59c700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2940a2cb0 0x7fb2940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.974+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2a00d0cd0 con 0x7fb2ac100820 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.974+0000 7fb293fff700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] conn(0x7fb2940ade10 0x7fb2940b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.974+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 --> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2a00043d0 con 0x7fb2940ade10 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.974+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb2a00d1120 con 0x7fb2ac100820 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.975+0000 7fb2aa59c700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2940a2cb0 0x7fb2940a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac107a40 tx=0x7fb29c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:26.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.975+0000 7fb2ab59e700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] conn(0x7fb2940ade10 0x7fb2940b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:26.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.975+0000 7fb2ab59e700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] conn(0x7fb2940ade10 0x7fb2940b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:26.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.976+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== osd.4 v2:172.21.15.111:6808/2127500568 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2a00043d0 con 0x7fb2940ade10 2024-04-22T01:11:26.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.992+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 --> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb2ac062e40 con 0x7fb2940ade10 2024-04-22T01:11:26.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb293fff700 1 -- 172.21.15.38:0/1973514043 <== osd.4 v2:172.21.15.111:6808/2127500568 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb2ac062e40 con 0x7fb2940ade10 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] conn(0x7fb2940ade10 msgr2=0x7fb2940b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] conn(0x7fb2940ade10 0x7fb2940b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2940a2cb0 msgr2=0x7fb2940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2940a2cb0 0x7fb2940a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac107a40 tx=0x7fb29c009260 comp rx=0 tx=0).stop 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac100820 msgr2=0x7fb2ac19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 --2- 172.21.15.38:0/1973514043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2ac100820 0x7fb2ac19e680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0000c00 tx=0x7fb2a000bab0 comp rx=0 tx=0).stop 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2ab59e700 1 -- 172.21.15.38:0/1973514043 reap_dead start 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 shutdown_connections 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.993+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 >> 172.21.15.38:0/1973514043 conn(0x7fb2ac0777a0 msgr2=0x7fb2ac0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.994+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 shutdown_connections 2024-04-22T01:11:26.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:26.994+0000 7fb2b1188700 1 -- 172.21.15.38:0/1973514043 wait complete. 2024-04-22T01:11:27.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:27 smithi111 ceph-mon[31057]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 215 B/s wr, 10 op/s 2024-04-22T01:11:27.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:27 smithi038 ceph-mon[28043]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 215 B/s wr, 10 op/s 2024-04-22T01:11:27.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:27 smithi203 ceph-mon[31825]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 215 B/s wr, 10 op/s 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 -- 172.21.15.38:0/960456160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c100780 msgr2=0x7f993c100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 --2- 172.21.15.38:0/960456160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c100780 0x7f993c100c00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f992c0099c0 tx=0x7f992c009cd0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 -- 172.21.15.38:0/960456160 shutdown_connections 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 --2- 172.21.15.38:0/960456160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f993c101140 0x7f993c10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 --2- 172.21.15.38:0/960456160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f993c107250 0x7f993c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 --2- 172.21.15.38:0/960456160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c100780 0x7f993c100c00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 -- 172.21.15.38:0/960456160 >> 172.21.15.38:0/960456160 conn(0x7f993c077730 msgr2=0x7f993c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.368+0000 7f99432ea700 1 -- 172.21.15.38:0/960456160 shutdown_connections 2024-04-22T01:11:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 -- 172.21.15.38:0/960456160 wait complete. 2024-04-22T01:11:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 Processor -- start 2024-04-22T01:11:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 -- start start 2024-04-22T01:11:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f993c100780 0x7f993c101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c101140 0x7f993c101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f993c107250 0x7f993c1023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f993c112730 con 0x7f993c101140 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1125b0 con 0x7f993c100780 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.369+0000 7f99432ea700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1128b0 con 0x7f993c107250 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c101140 0x7f993c101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c101140 0x7f993c101ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34428/0 (socket says 172.21.15.38:34428) 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 -- 172.21.15.38:0/3548780883 learned_addr learned my addr 172.21.15.38:0/3548780883 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9941086700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f993c100780 0x7f993c101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9941887700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f993c107250 0x7f993c1023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 -- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f993c107250 msgr2=0x7f993c1023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f993c107250 0x7f993c1023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 -- 172.21.15.38:0/3548780883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f993c100780 msgr2=0x7f993c101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f993c100780 0x7f993c101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 -- 172.21.15.38:0/3548780883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c1a7090 con 0x7f993c101140 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9941887700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f993c107250 0x7f993c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f9940885700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c101140 0x7f993c101ea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f992c006010 tx=0x7f992c0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c027070 con 0x7f993c101140 2024-04-22T01:11:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993c1a7320 con 0x7f993c101140 2024-04-22T01:11:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f992c004050 con 0x7f993c101140 2024-04-22T01:11:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.370+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c022e40 con 0x7f993c101140 2024-04-22T01:11:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.371+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993c1a7780 con 0x7f993c101140 2024-04-22T01:11:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.371+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f993c062940 con 0x7f993c101140 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f992c02a070 con 0x7f993c101140 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f99327fc700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99280a2cb0 0x7f99280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f992c0d0880 con 0x7f993c101140 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f9941086700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99280a2cb0 0x7f99280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f99327fc700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] conn(0x7f99280ade10 0x7f99280b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 --> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f992c001d30 con 0x7f99280ade10 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.373+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f992c0d0cd0 con 0x7f993c101140 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.374+0000 7f9941086700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99280a2cb0 0x7f99280a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9938009a10 tx=0x7f9938009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.374+0000 7f9941887700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] conn(0x7f99280ade10 0x7f99280b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.374+0000 7f9941887700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] conn(0x7f99280ade10 0x7f99280b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.375+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== osd.8 v2:172.21.15.203:6816/3416662120 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f992c001d30 con 0x7f99280ade10 2024-04-22T01:11:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.392+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 --> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f993c062d40 con 0x7f99280ade10 2024-04-22T01:11:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.392+0000 7f99327fc700 1 -- 172.21.15.38:0/3548780883 <== osd.8 v2:172.21.15.203:6816/3416662120 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f993c062d40 con 0x7f99280ade10 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.392+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] conn(0x7f99280ade10 msgr2=0x7f99280b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] conn(0x7f99280ade10 0x7f99280b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99280a2cb0 msgr2=0x7f99280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99280a2cb0 0x7f99280a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9938009a10 tx=0x7f9938009190 comp rx=0 tx=0).stop 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c101140 msgr2=0x7f993c101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 --2- 172.21.15.38:0/3548780883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f993c101140 0x7f993c101ea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f992c006010 tx=0x7f992c0038b0 comp rx=0 tx=0).stop 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f9941887700 1 -- 172.21.15.38:0/3548780883 reap_dead start 2024-04-22T01:11:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 shutdown_connections 2024-04-22T01:11:28.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 >> 172.21.15.38:0/3548780883 conn(0x7f993c077730 msgr2=0x7f993c1052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:28.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 shutdown_connections 2024-04-22T01:11:28.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:28.393+0000 7f99432ea700 1 -- 172.21.15.38:0/3548780883 wait complete. 2024-04-22T01:11:29.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:29 smithi111 ceph-mon[31057]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-22T01:11:29.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:29 smithi038 ceph-mon[28043]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-22T01:11:29.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:29 smithi203 ceph-mon[31825]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-22T01:11:31.002 INFO:teuthology.orchestra.run.smithi038.stdout:253403070505 2024-04-22T01:11:31.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-22T01:11:31.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:31 smithi038 ceph-mon[28043]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:31.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:31 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:11:31.626 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:31 smithi203 ceph-mon[31825]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:31.626 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:31 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:11:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:31 smithi111 ceph-mon[31057]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:31.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:31 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:11:31.773 INFO:teuthology.orchestra.run.smithi038.stdout:219043332141 2024-04-22T01:11:31.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T01:11:32.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 -- 172.21.15.38:0/2102976523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 msgr2=0x7feff010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 --2- 172.21.15.38:0/2102976523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff010f160 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b0d0 tx=0x7fefe800b3e0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 -- 172.21.15.38:0/2102976523 shutdown_connections 2024-04-22T01:11:32.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 --2- 172.21.15.38:0/2102976523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff010f160 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 --2- 172.21.15.38:0/2102976523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feff006ac40 0x7feff006b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 --2- 172.21.15.38:0/2102976523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff006b5f0 0x7feff0106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 -- 172.21.15.38:0/2102976523 >> 172.21.15.38:0/2102976523 conn(0x7feff00781d0 msgr2=0x7feff00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:32.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 -- 172.21.15.38:0/2102976523 shutdown_connections 2024-04-22T01:11:32.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 -- 172.21.15.38:0/2102976523 wait complete. 2024-04-22T01:11:32.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 Processor -- start 2024-04-22T01:11:32.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.573+0000 7feff647b700 1 -- start start 2024-04-22T01:11:32.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff647b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff006ac40 0x7feff019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff647b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feff006b5f0 0x7feff019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff647b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff0198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff647b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feff0112590 con 0x7feff0107590 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff647b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feff0112410 con 0x7feff006ac40 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff647b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feff0112710 con 0x7feff006b5f0 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff0198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff0198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34440/0 (socket says 172.21.15.38:34440) 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 -- 172.21.15.38:0/1447301936 learned_addr learned my addr 172.21.15.38:0/1447301936 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7fefeffff700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff006ac40 0x7feff019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7fefef7fe700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feff006b5f0 0x7feff019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 -- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feff006b5f0 msgr2=0x7feff019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feff006b5f0 0x7feff019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 -- 172.21.15.38:0/1447301936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff006ac40 msgr2=0x7feff019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff006ac40 0x7feff019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.574+0000 7feff4a18700 1 -- 172.21.15.38:0/1447301936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff0198ff0 con 0x7feff0107590 2024-04-22T01:11:32.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.575+0000 7feff4a18700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff0198810 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fefe8006070 tx=0x7fefe8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:32.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.575+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8021070 con 0x7feff0107590 2024-04-22T01:11:32.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.575+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff0199280 con 0x7feff0107590 2024-04-22T01:11:32.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.575+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe8007b80 con 0x7feff0107590 2024-04-22T01:11:32.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.575+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe80052a0 con 0x7feff0107590 2024-04-22T01:11:32.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.575+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff01a54a0 con 0x7feff0107590 2024-04-22T01:11:32.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.576+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feff00629c0 con 0x7feff0107590 2024-04-22T01:11:32.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.577+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fefe8024070 con 0x7feff0107590 2024-04-22T01:11:32.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.577+0000 7fefed7fa700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefdc0a2d30 0x7fefdc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefeffff700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefdc0a2d30 0x7fefdc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefe800f020 con 0x7feff0107590 2024-04-22T01:11:32.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefed7fa700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] conn(0x7fefdc0ade90 0x7fefdc0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 --> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefe8008ea0 con 0x7fefdc0ade90 2024-04-22T01:11:32.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fefe80d10c0 con 0x7feff0107590 2024-04-22T01:11:32.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefeffff700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefdc0a2d30 0x7fefdc0a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fefd8005950 tx=0x7fefd800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:32.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.578+0000 7fefef7fe700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] conn(0x7fefdc0ade90 0x7fefdc0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.579+0000 7fefef7fe700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] conn(0x7fefdc0ade90 0x7fefdc0b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:32.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.579+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== osd.5 v2:172.21.15.203:6808/3881588829 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fefe8008ea0 con 0x7fefdc0ade90 2024-04-22T01:11:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.596+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 --> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feff0062dc0 con 0x7fefdc0ade90 2024-04-22T01:11:32.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.596+0000 7fefed7fa700 1 -- 172.21.15.38:0/1447301936 <== osd.5 v2:172.21.15.203:6808/3881588829 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feff0062dc0 con 0x7fefdc0ade90 2024-04-22T01:11:32.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] conn(0x7fefdc0ade90 msgr2=0x7fefdc0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] conn(0x7fefdc0ade90 0x7fefdc0b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefdc0a2d30 msgr2=0x7fefdc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefdc0a2d30 0x7fefdc0a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fefd8005950 tx=0x7fefd800b410 comp rx=0 tx=0).stop 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 msgr2=0x7feff0198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 --2- 172.21.15.38:0/1447301936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feff0107590 0x7feff0198810 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fefe8006070 tx=0x7fefe8004d00 comp rx=0 tx=0).stop 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff4a18700 1 -- 172.21.15.38:0/1447301936 reap_dead start 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 shutdown_connections 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 >> 172.21.15.38:0/1447301936 conn(0x7feff00781d0 msgr2=0x7feff00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 shutdown_connections 2024-04-22T01:11:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.597+0000 7feff647b700 1 -- 172.21.15.38:0/1447301936 wait complete. 2024-04-22T01:11:32.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 -- 172.21.15.38:0/3687892595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 msgr2=0x7f4f501015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3687892595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 0x7f4f501015a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009990 tx=0x7f4f44009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 -- 172.21.15.38:0/3687892595 shutdown_connections 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3687892595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f50101ae0 0x7f4f5010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3687892595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f50107190 0x7f4f50107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3687892595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 0x7f4f501015a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 -- 172.21.15.38:0/3687892595 >> 172.21.15.38:0/3687892595 conn(0x7f4f500780a0 msgr2=0x7f4f500784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 -- 172.21.15.38:0/3687892595 shutdown_connections 2024-04-22T01:11:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.774+0000 7f4f54a64700 1 -- 172.21.15.38:0/3687892595 wait complete. 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 Processor -- start 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 -- start start 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 0x7f4f5019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f50101ae0 0x7f4f5019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f50107190 0x7f4f50197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4f50111400 con 0x7f4f50101120 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4f50111280 con 0x7f4f50101ae0 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f54a64700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f50111580 con 0x7f4f50107190 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f4e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 0x7f4f5019d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f4e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 0x7f4f5019d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34458/0 (socket says 172.21.15.38:34458) 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f4ed9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f50107190 0x7f4f50197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f4e59c700 1 -- 172.21.15.38:0/3051165403 learned_addr learned my addr 172.21.15.38:0/3051165403 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.775+0000 7f4f4dd9b700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f50101ae0 0x7f4f5019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4ed9d700 1 -- 172.21.15.38:0/3051165403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f50101ae0 msgr2=0x7f4f5019db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4ed9d700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f50101ae0 0x7f4f5019db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4ed9d700 1 -- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 msgr2=0x7f4f5019d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4ed9d700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f50101120 0x7f4f5019d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4ed9d700 1 -- 172.21.15.38:0/3051165403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f50197fe0 con 0x7f4f50107190 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4dd9b700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f50101ae0 0x7f4f5019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f4ed9d700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f50107190 0x7f4f50197700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800efc0 tx=0x7f4f4800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f480092e0 con 0x7f4f50107190 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f48004bb0 con 0x7f4f50107190 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48004280 con 0x7f4f50107190 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f50198270 con 0x7f4f50107190 2024-04-22T01:11:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.776+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f5006a970 con 0x7f4f50107190 2024-04-22T01:11:32.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.777+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f501985b0 con 0x7f4f50107190 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.778+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4f4801b030 con 0x7f4f50107190 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f3f7fe700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4f380a2d30 0x7f4f380a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f4e59c700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4f380a2d30 0x7f4f380a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4f480cfa20 con 0x7f4f50107190 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f3f7fe700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] conn(0x7f4f380ade90 0x7f4f380b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 --> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f4801fe50 con 0x7f4f380ade90 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f4f480cfdd0 con 0x7f4f50107190 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f4dd9b700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] conn(0x7f4f380ade90 0x7f4f380b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.779+0000 7f4f4e59c700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4f380a2d30 0x7f4f380a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009ca0 tx=0x7f4f40010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:32.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.780+0000 7f4f4dd9b700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] conn(0x7f4f380ade90 0x7f4f380b0330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:32.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.780+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== osd.0 v2:172.21.15.38:6802/3120557304 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f4801fe50 con 0x7f4f380ade90 2024-04-22T01:11:32.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.797+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 --> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4f5006b100 con 0x7f4f380ade90 2024-04-22T01:11:32.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.797+0000 7f4f3f7fe700 1 -- 172.21.15.38:0/3051165403 <== osd.0 v2:172.21.15.38:6802/3120557304 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4f5006b100 con 0x7f4f380ade90 2024-04-22T01:11:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.797+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] conn(0x7f4f380ade90 msgr2=0x7f4f380b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.797+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] conn(0x7f4f380ade90 0x7f4f380b0330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4f380a2d30 msgr2=0x7f4f380a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4f380a2d30 0x7f4f380a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009ca0 tx=0x7f4f40010040 comp rx=0 tx=0).stop 2024-04-22T01:11:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f50107190 msgr2=0x7f4f50197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:32.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 --2- 172.21.15.38:0/3051165403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f50107190 0x7f4f50197700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800efc0 tx=0x7f4f4800c3a0 comp rx=0 tx=0).stop 2024-04-22T01:11:32.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f4ed9d700 1 -- 172.21.15.38:0/3051165403 reap_dead start 2024-04-22T01:11:32.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 shutdown_connections 2024-04-22T01:11:32.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 >> 172.21.15.38:0/3051165403 conn(0x7f4f500780a0 msgr2=0x7f4f500ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:32.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 shutdown_connections 2024-04-22T01:11:32.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:32.798+0000 7f4f54a64700 1 -- 172.21.15.38:0/3051165403 wait complete. 2024-04-22T01:11:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:33 smithi038 ceph-mon[28043]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:33 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:33 smithi111 ceph-mon[31057]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:33.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:33 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:33 smithi203 ceph-mon[31825]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:33.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.767 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:33 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:11:33.790 INFO:teuthology.orchestra.run.smithi038.stdout:163208757305 2024-04-22T01:11:33.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T01:11:34.984 INFO:teuthology.orchestra.run.smithi038.stdout:34359738451 2024-04-22T01:11:34.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T01:11:35.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 -- 172.21.15.38:0/1183069255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb54069f40 msgr2=0x7fcb5406a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 --2- 172.21.15.38:0/1183069255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb54069f40 0x7fcb5406a320 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009990 tx=0x7fcb44009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 -- 172.21.15.38:0/1183069255 shutdown_connections 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 --2- 172.21.15.38:0/1183069255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5410ece0 0x7fcb541110d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 --2- 172.21.15.38:0/1183069255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb54069f40 0x7fcb5406a320 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 --2- 172.21.15.38:0/1183069255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb5406a860 0x7fcb5410e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 -- 172.21.15.38:0/1183069255 >> 172.21.15.38:0/1183069255 conn(0x7fcb540780e0 msgr2=0x7fcb540784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 -- 172.21.15.38:0/1183069255 shutdown_connections 2024-04-22T01:11:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.186+0000 7fcb5883c700 1 -- 172.21.15.38:0/1183069255 wait complete. 2024-04-22T01:11:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 Processor -- start 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 -- start start 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb54069f40 0x7fcb541a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5406a860 0x7fcb541a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5410ece0 0x7fcb5419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54114760 con 0x7fcb5410ece0 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541145e0 con 0x7fcb54069f40 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb5883c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541148e0 con 0x7fcb5406a860 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb52d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5410ece0 0x7fcb5419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb51d9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5406a860 0x7fcb541a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb51d9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5406a860 0x7fcb541a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51140/0 (socket says 172.21.15.38:51140) 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb52d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5410ece0 0x7fcb5419aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34476/0 (socket says 172.21.15.38:34476) 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.187+0000 7fcb51d9b700 1 -- 172.21.15.38:0/3871623174 learned_addr learned my addr 172.21.15.38:0/3871623174 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb5259c700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb54069f40 0x7fcb541a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb51d9b700 1 -- 172.21.15.38:0/3871623174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb54069f40 msgr2=0x7fcb541a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb51d9b700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb54069f40 0x7fcb541a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb51d9b700 1 -- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5410ece0 msgr2=0x7fcb5419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb51d9b700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5410ece0 0x7fcb5419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb51d9b700 1 -- 172.21.15.38:0/3871623174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5419b240 con 0x7fcb5406a860 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb52d9d700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5410ece0 0x7fcb5419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb51d9b700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5406a860 0x7fcb541a0e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800e9a0 tx=0x7fcb4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4800cbd0 con 0x7fcb5406a860 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5419b530 con 0x7fcb5406a860 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb541a77f0 con 0x7fcb5406a860 2024-04-22T01:11:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb4800cd70 con 0x7fcb5406a860 2024-04-22T01:11:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.188+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48018680 con 0x7fcb5406a860 2024-04-22T01:11:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.189+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb34001030 con 0x7fcb5406a860 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.191+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcb480188e0 con 0x7fcb5406a860 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb437fe700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcb3c0a2d30 0x7fcb3c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcb480d01e0 con 0x7fcb5406a860 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb437fe700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] conn(0x7fcb3c0ade90 0x7fcb3c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 --> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb4800e1a0 con 0x7fcb3c0ade90 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fcb480d0630 con 0x7fcb5406a860 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb5259c700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcb3c0a2d30 0x7fcb3c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.192+0000 7fcb52d9d700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] conn(0x7fcb3c0ade90 0x7fcb3c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.193+0000 7fcb52d9d700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] conn(0x7fcb3c0ade90 0x7fcb3c0b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.193+0000 7fcb5259c700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcb3c0a2d30 0x7fcb3c0a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb44006010 tx=0x7fcb4401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.194+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== osd.11 v2:172.21.15.203:6824/3028852756 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb4800e1a0 con 0x7fcb3c0ade90 2024-04-22T01:11:35.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.214+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 --> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb34002e60 con 0x7fcb3c0ade90 2024-04-22T01:11:35.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb437fe700 1 -- 172.21.15.38:0/3871623174 <== osd.11 v2:172.21.15.203:6824/3028852756 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcb34002e60 con 0x7fcb3c0ade90 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] conn(0x7fcb3c0ade90 msgr2=0x7fcb3c0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] conn(0x7fcb3c0ade90 0x7fcb3c0b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcb3c0a2d30 msgr2=0x7fcb3c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcb3c0a2d30 0x7fcb3c0a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb44006010 tx=0x7fcb4401a040 comp rx=0 tx=0).stop 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5406a860 msgr2=0x7fcb541a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 --2- 172.21.15.38:0/3871623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb5406a860 0x7fcb541a0e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800e9a0 tx=0x7fcb4800ecb0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb52d9d700 1 -- 172.21.15.38:0/3871623174 reap_dead start 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 shutdown_connections 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 >> 172.21.15.38:0/3871623174 conn(0x7fcb540780e0 msgr2=0x7fcb54110230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.215+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 shutdown_connections 2024-04-22T01:11:35.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.216+0000 7fcb5883c700 1 -- 172.21.15.38:0/3871623174 wait complete. 2024-04-22T01:11:35.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:35 smithi038 ceph-mon[28043]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/2731423711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f641082b0 msgr2=0x7f9f64108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/2731423711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f641082b0 0x7f9f64108690 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009990 tx=0x7f9f60009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/2731423711 shutdown_connections 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/2731423711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f6406a8e0 0x7f9f6410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/2731423711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f641082b0 0x7f9f64108690 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/2731423711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f64069f40 0x7f9f6406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/2731423711 >> 172.21.15.38:0/2731423711 conn(0x7f9f640780e0 msgr2=0x7f9f640784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:35.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/2731423711 shutdown_connections 2024-04-22T01:11:35.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.528+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/2731423711 wait complete. 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 Processor -- start 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 -- start start 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f64069f40 0x7f9f641a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f6406a8e0 0x7f9f641a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f641082b0 0x7f9f6419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641124f0 con 0x7f9f6406a8e0 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64112370 con 0x7f9f641082b0 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f6b7e6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64112670 con 0x7f9f64069f40 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f68d81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f6406a8e0 0x7f9f641a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f68d81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f6406a8e0 0x7f9f641a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34494/0 (socket says 172.21.15.38:34494) 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f69582700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f64069f40 0x7f9f641a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.529+0000 7f9f68d81700 1 -- 172.21.15.38:0/4106141361 learned_addr learned my addr 172.21.15.38:0/4106141361 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f69d83700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f641082b0 0x7f9f6419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f68d81700 1 -- 172.21.15.38:0/4106141361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f64069f40 msgr2=0x7f9f641a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f68d81700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f64069f40 0x7f9f641a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f68d81700 1 -- 172.21.15.38:0/4106141361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f641082b0 msgr2=0x7f9f6419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f68d81700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f641082b0 0x7f9f6419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f68d81700 1 -- 172.21.15.38:0/4106141361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6419b180 con 0x7f9f6406a8e0 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f68d81700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f6406a8e0 0x7f9f641a0d50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400e9a0 tx=0x7f9f5400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5400ca10 con 0x7f9f6406a8e0 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6419b470 con 0x7f9f6406a8e0 2024-04-22T01:11:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f5400cbb0 con 0x7f9f6406a8e0 2024-04-22T01:11:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54018680 con 0x7f9f6406a8e0 2024-04-22T01:11:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.530+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f641a76e0 con 0x7f9f6406a8e0 2024-04-22T01:11:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.531+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f641005f0 con 0x7f9f6406a8e0 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.533+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9f54010430 con 0x7f9f6406a8e0 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.533+0000 7f9f5a7fc700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9f500a2cb0 0x7f9f500a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.533+0000 7f9f69582700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9f500a2cb0 0x7f9f500a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.533+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f540cff50 con 0x7f9f6406a8e0 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.534+0000 7f9f5a7fc700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] conn(0x7f9f500ade10 0x7f9f500b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.534+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 --> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f5401bec0 con 0x7f9f500ade10 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.534+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9f540d03a0 con 0x7f9f6406a8e0 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.534+0000 7f9f69582700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9f500a2cb0 0x7f9f500a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f640ff760 tx=0x7f9f60005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.534+0000 7f9f69d83700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] conn(0x7f9f500ade10 0x7f9f500b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.534+0000 7f9f69d83700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] conn(0x7f9f500ade10 0x7f9f500b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.535+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== osd.6 v2:172.21.15.38:6818/3905664243 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f5401bec0 con 0x7f9f500ade10 2024-04-22T01:11:35.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.552+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 --> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f64103960 con 0x7f9f500ade10 2024-04-22T01:11:35.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f5a7fc700 1 -- 172.21.15.38:0/4106141361 <== osd.6 v2:172.21.15.38:6818/3905664243 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9f64103960 con 0x7f9f500ade10 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] conn(0x7f9f500ade10 msgr2=0x7f9f500b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] conn(0x7f9f500ade10 0x7f9f500b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9f500a2cb0 msgr2=0x7f9f500a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9f500a2cb0 0x7f9f500a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f640ff760 tx=0x7f9f60005960 comp rx=0 tx=0).stop 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f6406a8e0 msgr2=0x7f9f641a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:35.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 --2- 172.21.15.38:0/4106141361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f6406a8e0 0x7f9f641a0d50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400e9a0 tx=0x7f9f5400ecb0 comp rx=0 tx=0).stop 2024-04-22T01:11:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f69d83700 1 -- 172.21.15.38:0/4106141361 reap_dead start 2024-04-22T01:11:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 shutdown_connections 2024-04-22T01:11:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.553+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 >> 172.21.15.38:0/4106141361 conn(0x7f9f640780e0 msgr2=0x7f9f640ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.554+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 shutdown_connections 2024-04-22T01:11:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:35.554+0000 7f9f6b7e6700 1 -- 172.21.15.38:0/4106141361 wait complete. 2024-04-22T01:11:35.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:35 smithi203 ceph-mon[31825]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:11:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:35 smithi111 ceph-mon[31057]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:11:36.332 INFO:teuthology.orchestra.run.smithi038.stdout:141733920830 2024-04-22T01:11:36.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.526+0000 7fb5b78a3700 1 -- 172.21.15.38:0/2492800177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b01071a0 msgr2=0x7fb5b0107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.526+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/2492800177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b01071a0 0x7fb5b0107580 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009990 tx=0x7fb5ac009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 -- 172.21.15.38:0/2492800177 shutdown_connections 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/2492800177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101af0 0x7fb5b010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/2492800177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b01071a0 0x7fb5b0107580 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/2492800177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b0101130 0x7fb5b01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 -- 172.21.15.38:0/2492800177 >> 172.21.15.38:0/2492800177 conn(0x7fb5b00780e0 msgr2=0x7fb5b00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:36.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 -- 172.21.15.38:0/2492800177 shutdown_connections 2024-04-22T01:11:36.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 -- 172.21.15.38:0/2492800177 wait complete. 2024-04-22T01:11:36.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.527+0000 7fb5b78a3700 1 Processor -- start 2024-04-22T01:11:36.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 -- start start 2024-04-22T01:11:36.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b0101130 0x7fb5b010ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b0101af0 0x7fb5b010efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b01071a0 0x7fb5b010f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b0112550 con 0x7fb5b0101130 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b01123d0 con 0x7fb5b01071a0 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b78a3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b01126d0 con 0x7fb5b0101af0 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b0101130 0x7fb5b010ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b0101130 0x7fb5b010ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34516/0 (socket says 172.21.15.38:34516) 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 -- 172.21.15.38:0/3480391539 learned_addr learned my addr 172.21.15.38:0/3480391539 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b4e3e700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b0101af0 0x7fb5b010efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 -- 172.21.15.38:0/3480391539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b0101af0 msgr2=0x7fb5b010efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b0101af0 0x7fb5b010efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 -- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b01071a0 msgr2=0x7fb5b010f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b01071a0 0x7fb5b010f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.528+0000 7fb5b563f700 1 -- 172.21.15.38:0/3480391539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b01a6f60 con 0x7fb5b0101130 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.529+0000 7fb5b563f700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b0101130 0x7fb5b010ea70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac000c00 tx=0x7fb5ac003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:36.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.529+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac027070 con 0x7fb5b0101130 2024-04-22T01:11:36.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.529+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b01a71f0 con 0x7fb5b0101130 2024-04-22T01:11:36.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.529+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5ac004190 con 0x7fb5b0101130 2024-04-22T01:11:36.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.529+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00f460 con 0x7fb5b0101130 2024-04-22T01:11:36.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.529+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b01a76a0 con 0x7fb5b0101130 2024-04-22T01:11:36.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.530+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb5b0062940 con 0x7fb5b0101130 2024-04-22T01:11:36.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.531+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb5ac02a070 con 0x7fb5b0101130 2024-04-22T01:11:36.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5a67fc700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb59c0a2d30 0x7fb59c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5b4e3e700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb59c0a2d30 0x7fb59c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb5ac0d0b20 con 0x7fb5b0101130 2024-04-22T01:11:36.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5a67fc700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] conn(0x7fb59c0ade90 0x7fb59c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 --> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5ac00b0b0 con 0x7fb59c0ade90 2024-04-22T01:11:36.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb5ac0d0f70 con 0x7fb5b0101130 2024-04-22T01:11:36.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.532+0000 7fb5b4e3e700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb59c0a2d30 0x7fb59c0a51f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00786e0 tx=0x7fb5a0005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:36.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.533+0000 7fb5b5e40700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] conn(0x7fb59c0ade90 0x7fb59c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.533+0000 7fb5b5e40700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] conn(0x7fb59c0ade90 0x7fb59c0b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:36.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.534+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== osd.1 v2:172.21.15.111:6800/3032046140 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5ac00b0b0 con 0x7fb59c0ade90 2024-04-22T01:11:36.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.551+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 --> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb5b0062d40 con 0x7fb59c0ade90 2024-04-22T01:11:36.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.552+0000 7fb5a67fc700 1 -- 172.21.15.38:0/3480391539 <== osd.1 v2:172.21.15.111:6800/3032046140 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb5b0062d40 con 0x7fb59c0ade90 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.552+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] conn(0x7fb59c0ade90 msgr2=0x7fb59c0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.552+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] conn(0x7fb59c0ade90 0x7fb59c0b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb59c0a2d30 msgr2=0x7fb59c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb59c0a2d30 0x7fb59c0a51f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00786e0 tx=0x7fb5a0005eb0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b0101130 msgr2=0x7fb5b010ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 --2- 172.21.15.38:0/3480391539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b0101130 0x7fb5b010ea70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac000c00 tx=0x7fb5ac003730 comp rx=0 tx=0).stop 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b5e40700 1 -- 172.21.15.38:0/3480391539 reap_dead start 2024-04-22T01:11:36.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 shutdown_connections 2024-04-22T01:11:36.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 >> 172.21.15.38:0/3480391539 conn(0x7fb5b00780e0 msgr2=0x7fb5b00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:36.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 shutdown_connections 2024-04-22T01:11:36.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.553+0000 7fb5b78a3700 1 -- 172.21.15.38:0/3480391539 wait complete. 2024-04-22T01:11:36.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 -- 172.21.15.38:0/947812749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 msgr2=0x7eff3410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 --2- 172.21.15.38:0/947812749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff3410f140 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00b0a0 tx=0x7eff2c00b3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 -- 172.21.15.38:0/947812749 shutdown_connections 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 --2- 172.21.15.38:0/947812749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff3410f140 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 --2- 172.21.15.38:0/947812749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff341024e0 0x7eff341028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 --2- 172.21.15.38:0/947812749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff34102e90 0x7eff34106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 -- 172.21.15.38:0/947812749 >> 172.21.15.38:0/947812749 conn(0x7eff34079390 msgr2=0x7eff340797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 -- 172.21.15.38:0/947812749 shutdown_connections 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 -- 172.21.15.38:0/947812749 wait complete. 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 Processor -- start 2024-04-22T01:11:36.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.740+0000 7eff3b647700 1 -- start start 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff3b647700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff341024e0 0x7eff3419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff3b647700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff34102e90 0x7eff3419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff3b647700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff34198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff3b647700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff34112550 con 0x7eff34107570 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff3b647700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7eff341123d0 con 0x7eff341024e0 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff3b647700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff341126d0 con 0x7eff34102e90 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff34198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff34198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35060/0 (socket says 172.21.15.38:35060) 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 -- 172.21.15.38:0/3936551933 learned_addr learned my addr 172.21.15.38:0/3936551933 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff393e3700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff341024e0 0x7eff3419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff38be2700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff34102e90 0x7eff3419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 -- 172.21.15.38:0/3936551933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff34102e90 msgr2=0x7eff3419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff34102e90 0x7eff3419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 -- 172.21.15.38:0/3936551933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff341024e0 msgr2=0x7eff3419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff341024e0 0x7eff3419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 -- 172.21.15.38:0/3936551933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff34199050 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff39be4700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff34198770 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0062a0 tx=0x7eff2c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.741+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c021070 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.742+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff341992e0 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.742+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff2c007c80 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.742+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c005420 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.742+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff341a55d0 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.742+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff34062980 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.744+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7eff2c024070 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.744+0000 7eff267fc700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff200a2cb0 0x7eff200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff393e3700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff200a2cb0 0x7eff200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7eff2c00f020 con 0x7eff34107570 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff267fc700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] conn(0x7eff200ade10 0x7eff200b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 --> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff2c012e50 con 0x7eff200ade10 2024-04-22T01:11:36.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7eff2c0d0a70 con 0x7eff34107570 2024-04-22T01:11:36.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff38be2700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] conn(0x7eff200ade10 0x7eff200b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:36.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff393e3700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff200a2cb0 0x7eff200a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7eff30005950 tx=0x7eff3000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:36.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.745+0000 7eff38be2700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] conn(0x7eff200ade10 0x7eff200b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:36.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.746+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== osd.3 v2:172.21.15.38:6810/3150547332 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff2c012e50 con 0x7eff200ade10 2024-04-22T01:11:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.762+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 --> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7eff34062d80 con 0x7eff200ade10 2024-04-22T01:11:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff267fc700 1 -- 172.21.15.38:0/3936551933 <== osd.3 v2:172.21.15.38:6810/3150547332 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7eff34062d80 con 0x7eff200ade10 2024-04-22T01:11:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] conn(0x7eff200ade10 msgr2=0x7eff200b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] conn(0x7eff200ade10 0x7eff200b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff200a2cb0 msgr2=0x7eff200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff200a2cb0 0x7eff200a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7eff30005950 tx=0x7eff3000a300 comp rx=0 tx=0).stop 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 msgr2=0x7eff34198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 --2- 172.21.15.38:0/3936551933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff34107570 0x7eff34198770 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0062a0 tx=0x7eff2c00be60 comp rx=0 tx=0).stop 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff39be4700 1 -- 172.21.15.38:0/3936551933 reap_dead start 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 shutdown_connections 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 >> 172.21.15.38:0/3936551933 conn(0x7eff34079390 msgr2=0x7eff341002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 shutdown_connections 2024-04-22T01:11:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:36.763+0000 7eff3b647700 1 -- 172.21.15.38:0/3936551933 wait complete. 2024-04-22T01:11:37.405 INFO:teuthology.orchestra.run.smithi038.stdout:343597383707 2024-04-22T01:11:37.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-22T01:11:37.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:37 smithi038 ceph-mon[28043]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-22T01:11:37.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:37 smithi203 ceph-mon[31825]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-22T01:11:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:37 smithi111 ceph-mon[31057]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-22T01:11:38.721 INFO:teuthology.orchestra.run.smithi038.stdout:197568495669 2024-04-22T01:11:38.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T01:11:39.604 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:39 smithi038 ceph-mon[28043]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:11:39.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 -- 172.21.15.38:0/466005588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be4107570 msgr2=0x7f0be410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 --2- 172.21.15.38:0/466005588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be4107570 0x7f0be410f140 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00b0a0 tx=0x7f0bdc00b3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 -- 172.21.15.38:0/466005588 shutdown_connections 2024-04-22T01:11:39.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 --2- 172.21.15.38:0/466005588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be4107570 0x7f0be410f140 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 --2- 172.21.15.38:0/466005588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0be41024e0 0x7f0be41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 --2- 172.21.15.38:0/466005588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be4102e90 0x7f0be4106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 -- 172.21.15.38:0/466005588 >> 172.21.15.38:0/466005588 conn(0x7f0be4079390 msgr2=0x7f0be40797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:39.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 -- 172.21.15.38:0/466005588 shutdown_connections 2024-04-22T01:11:39.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 -- 172.21.15.38:0/466005588 wait complete. 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.605+0000 7f0be901d700 1 Processor -- start 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 -- start start 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be41024e0 0x7f0be419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0be4102e90 0x7f0be419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be4107570 0x7f0be41987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0be4112550 con 0x7f0be41024e0 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41123d0 con 0x7f0be4102e90 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be901d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41126d0 con 0x7f0be4107570 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be41024e0 0x7f0be419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be41024e0 0x7f0be419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35082/0 (socket says 172.21.15.38:35082) 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be2d9d700 1 -- 172.21.15.38:0/1476178767 learned_addr learned my addr 172.21.15.38:0/1476178767 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be359e700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be4107570 0x7f0be41987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be259c700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0be4102e90 0x7f0be419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be2d9d700 1 -- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be4107570 msgr2=0x7f0be41987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.606+0000 7f0be2d9d700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be4107570 0x7f0be41987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0be2d9d700 1 -- 172.21.15.38:0/1476178767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0be4102e90 msgr2=0x7f0be419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0be2d9d700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0be4102e90 0x7f0be419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0be2d9d700 1 -- 172.21.15.38:0/1476178767 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be4198f90 con 0x7f0be41024e0 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0be2d9d700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be41024e0 0x7f0be419e630 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400d950 tx=0x7f0bd400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4016070 con 0x7f0be41024e0 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bd400fd70 con 0x7f0be41024e0 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be4199280 con 0x7f0be41024e0 2024-04-22T01:11:39.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400e610 con 0x7f0be41024e0 2024-04-22T01:11:39.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.607+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be41a5490 con 0x7f0be41024e0 2024-04-22T01:11:39.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.608+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0be40629c0 con 0x7f0be41024e0 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.609+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0bd4019070 con 0x7f0be41024e0 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0bcbfff700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0bcc0a2cb0 0x7f0bcc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0be259c700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0bcc0a2cb0 0x7f0bcc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0bd40cf5d0 con 0x7f0be41024e0 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0bcbfff700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] conn(0x7f0bcc0ade10 0x7f0bcc0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 --> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bd4027ea0 con 0x7f0bcc0ade10 2024-04-22T01:11:39.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0bd40cf980 con 0x7f0be41024e0 2024-04-22T01:11:39.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0be359e700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] conn(0x7f0bcc0ade10 0x7f0bcc0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.610+0000 7f0be259c700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0bcc0a2cb0 0x7f0bcc0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0be4106ff0 tx=0x7f0bd800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:39.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.611+0000 7f0be359e700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] conn(0x7f0bcc0ade10 0x7f0bcc0b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:39.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.612+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== osd.2 v2:172.21.15.203:6800/1524209627 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bd4027ea0 con 0x7f0bcc0ade10 2024-04-22T01:11:39.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.628+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 --> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0be4062dc0 con 0x7f0bcc0ade10 2024-04-22T01:11:39.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0bcbfff700 1 -- 172.21.15.38:0/1476178767 <== osd.2 v2:172.21.15.203:6800/1524209627 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0be4062dc0 con 0x7f0bcc0ade10 2024-04-22T01:11:39.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] conn(0x7f0bcc0ade10 msgr2=0x7f0bcc0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be901d700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] conn(0x7f0bcc0ade10 0x7f0bcc0b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0bcc0a2cb0 msgr2=0x7f0bcc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be901d700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0bcc0a2cb0 0x7f0bcc0a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0be4106ff0 tx=0x7f0bd800b410 comp rx=0 tx=0).stop 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be41024e0 msgr2=0x7f0be419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be901d700 1 --2- 172.21.15.38:0/1476178767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0be41024e0 0x7f0be419e630 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400d950 tx=0x7f0bd400dc60 comp rx=0 tx=0).stop 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.629+0000 7f0be359e700 1 -- 172.21.15.38:0/1476178767 reap_dead start 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.630+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 shutdown_connections 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.630+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 >> 172.21.15.38:0/1476178767 conn(0x7f0be4079390 msgr2=0x7f0be4101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.630+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 shutdown_connections 2024-04-22T01:11:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.630+0000 7f0be901d700 1 -- 172.21.15.38:0/1476178767 wait complete. 2024-04-22T01:11:39.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:39 smithi203 ceph-mon[31825]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 -- 172.21.15.38:0/864847942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25641064b0 msgr2=0x7f256410e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 --2- 172.21.15.38:0/864847942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25641064b0 0x7f256410e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f255c00b0d0 tx=0x7f255c00b3e0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 -- 172.21.15.38:0/864847942 shutdown_connections 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 --2- 172.21.15.38:0/864847942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25641064b0 0x7f256410e0c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 --2- 172.21.15.38:0/864847942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2564101b90 0x7f2564101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 --2- 172.21.15.38:0/864847942 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2564102540 0x7f2564105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 -- 172.21.15.38:0/864847942 >> 172.21.15.38:0/864847942 conn(0x7f25640777a0 msgr2=0x7f25640feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 -- 172.21.15.38:0/864847942 shutdown_connections 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.825+0000 7f256c44d700 1 -- 172.21.15.38:0/864847942 wait complete. 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 Processor -- start 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 -- start start 2024-04-22T01:11:39.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2564101b90 0x7f256419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2564102540 0x7f256419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25641064b0 0x7f25641988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25641114f0 con 0x7f2564102540 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2564111370 con 0x7f25641064b0 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256c44d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2564111670 con 0x7f2564101b90 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2564102540 0x7f256419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2564102540 0x7f256419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35112/0 (socket says 172.21.15.38:35112) 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 -- 172.21.15.38:0/601413700 learned_addr learned my addr 172.21.15.38:0/601413700 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256a9ea700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25641064b0 0x7f25641988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256a1e9700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2564101b90 0x7f256419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 -- 172.21.15.38:0/601413700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2564101b90 msgr2=0x7f256419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2564101b90 0x7f256419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 -- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25641064b0 msgr2=0x7f25641988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25641064b0 0x7f25641988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f25699e8700 1 -- 172.21.15.38:0/601413700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2564199020 con 0x7f2564102540 2024-04-22T01:11:39.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.826+0000 7f256a9ea700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25641064b0 0x7f25641988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:39.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.827+0000 7f25699e8700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2564102540 0x7f256419ec40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f255400d980 tx=0x7f255400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:39.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.827+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554017070 con 0x7f2564102540 2024-04-22T01:11:39.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.827+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2564199310 con 0x7f2564102540 2024-04-22T01:11:39.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.827+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2554009b40 con 0x7f2564102540 2024-04-22T01:11:39.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.827+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255400f620 con 0x7f2564102540 2024-04-22T01:11:39.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.827+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25641a55d0 con 0x7f2564102540 2024-04-22T01:11:39.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.828+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2564062940 con 0x7f2564102540 2024-04-22T01:11:39.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.830+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2554009cf0 con 0x7f2564102540 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.831+0000 7f255b7fe700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f25500a2cb0 0x7f25500a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.831+0000 7f256a1e9700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f25500a2cb0 0x7f25500a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.831+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f25540cf470 con 0x7f2564102540 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.831+0000 7f255b7fe700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] conn(0x7f25500ade10 0x7f25500b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.831+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 --> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2554020e70 con 0x7f25500ade10 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.831+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f25540cf820 con 0x7f2564102540 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.832+0000 7f256a1e9700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f25500a2cb0 0x7f25500a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2560005950 tx=0x7f2560005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.832+0000 7f256a9ea700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] conn(0x7f25500ade10 0x7f25500b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:39.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.832+0000 7f256a9ea700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] conn(0x7f25500ade10 0x7f25500b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:39.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.833+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== osd.10 v2:172.21.15.111:6824/1666422758 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2554020e70 con 0x7f25500ade10 2024-04-22T01:11:39.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.850+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 --> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2564062d40 con 0x7f25500ade10 2024-04-22T01:11:39.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f255b7fe700 1 -- 172.21.15.38:0/601413700 <== osd.10 v2:172.21.15.111:6824/1666422758 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2564062d40 con 0x7f25500ade10 2024-04-22T01:11:39.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] conn(0x7f25500ade10 msgr2=0x7f25500b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] conn(0x7f25500ade10 0x7f25500b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f25500a2cb0 msgr2=0x7f25500a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f25500a2cb0 0x7f25500a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2560005950 tx=0x7f2560005ed0 comp rx=0 tx=0).stop 2024-04-22T01:11:39.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2564102540 msgr2=0x7f256419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:39.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 --2- 172.21.15.38:0/601413700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2564102540 0x7f256419ec40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f255400d980 tx=0x7f255400dd40 comp rx=0 tx=0).stop 2024-04-22T01:11:39.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256a9ea700 1 -- 172.21.15.38:0/601413700 reap_dead start 2024-04-22T01:11:39.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 shutdown_connections 2024-04-22T01:11:39.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.851+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 >> 172.21.15.38:0/601413700 conn(0x7f25640777a0 msgr2=0x7f2564100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:39.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.852+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 shutdown_connections 2024-04-22T01:11:39.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:39.852+0000 7f256c44d700 1 -- 172.21.15.38:0/601413700 wait complete. 2024-04-22T01:11:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:39 smithi111 ceph-mon[31057]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:11:40.509 INFO:teuthology.orchestra.run.smithi038.stdout:77309411403 2024-04-22T01:11:40.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T01:11:41.153 INFO:teuthology.orchestra.run.smithi038.stdout:107374182469 2024-04-22T01:11:41.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T01:11:41.600 INFO:teuthology.orchestra.run.smithi038.stdout:55834574927 2024-04-22T01:11:41.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T01:11:41.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:41 smithi203 ceph-mon[31825]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:41.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:41 smithi038 ceph-mon[28043]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:41 smithi111 ceph-mon[31057]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:42.058 INFO:teuthology.orchestra.run.smithi038.stdout:309237645343 2024-04-22T01:11:42.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-22T01:11:42.279 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:42.279 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:42.280 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:42.283 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:42.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.305+0000 7f5b9d039700 1 -- 172.21.15.38:0/4191246440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98110a10 msgr2=0x7f5b98109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:42.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.305+0000 7f5b9d039700 1 --2- 172.21.15.38:0/4191246440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98110a10 0x7f5b98109230 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5b880099e0 tx=0x7f5b88009cf0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.308+0000 7f5b9d039700 1 -- 172.21.15.38:0/4191246440 shutdown_connections 2024-04-22T01:11:42.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.308+0000 7f5b9d039700 1 --2- 172.21.15.38:0/4191246440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b9811ba10 0x7f5b9811de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.308+0000 7f5b9d039700 1 --2- 172.21.15.38:0/4191246440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98110a10 0x7f5b98109230 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.308+0000 7f5b9d039700 1 --2- 172.21.15.38:0/4191246440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b98109770 0x7f5b98109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.308+0000 7f5b9d039700 1 -- 172.21.15.38:0/4191246440 >> 172.21.15.38:0/4191246440 conn(0x7f5b9806d3a0 msgr2=0x7f5b9806d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:42.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.309+0000 7f5b9d039700 1 -- 172.21.15.38:0/4191246440 shutdown_connections 2024-04-22T01:11:42.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.309+0000 7f5b9d039700 1 -- 172.21.15.38:0/4191246440 wait complete. 2024-04-22T01:11:42.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 Processor -- start 2024-04-22T01:11:42.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 -- start start 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98109770 0x7f5b9811b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b98110a10 0x7f5b98115190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b9811ba10 0x7f5b981156d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98111c40 con 0x7f5b98109770 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98111ac0 con 0x7f5b98110a10 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.311+0000 7f5b9d039700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98111dc0 con 0x7f5b9811ba10 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.312+0000 7f5b9759e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b9811ba10 0x7f5b981156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.312+0000 7f5b9759e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b9811ba10 0x7f5b981156d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39042/0 (socket says 172.21.15.38:39042) 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.312+0000 7f5b9759e700 1 -- 172.21.15.38:0/871338797 learned_addr learned my addr 172.21.15.38:0/871338797 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.312+0000 7f5b96d9d700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98109770 0x7f5b9811b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.313+0000 7f5b9659c700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b98110a10 0x7f5b98115190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.313+0000 7f5b96d9d700 1 -- 172.21.15.38:0/871338797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b9811ba10 msgr2=0x7f5b981156d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.313+0000 7f5b96d9d700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b9811ba10 0x7f5b981156d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.313+0000 7f5b96d9d700 1 -- 172.21.15.38:0/871338797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b98110a10 msgr2=0x7f5b98115190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:42.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.313+0000 7f5b96d9d700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b98110a10 0x7f5b98115190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.313+0000 7f5b96d9d700 1 -- 172.21.15.38:0/871338797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b98115e00 con 0x7f5b98109770 2024-04-22T01:11:42.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.314+0000 7f5b96d9d700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98109770 0x7f5b9811b110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5b88006010 tx=0x7f5b8800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:42.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.316+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b88026070 con 0x7f5b98109770 2024-04-22T01:11:42.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.316+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9807a490 con 0x7f5b98109770 2024-04-22T01:11:42.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.316+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b9807a920 con 0x7f5b98109770 2024-04-22T01:11:42.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.316+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b88004160 con 0x7f5b98109770 2024-04-22T01:11:42.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.316+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b88021410 con 0x7f5b98109770 2024-04-22T01:11:42.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.319+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b84001030 con 0x7f5b98109770 2024-04-22T01:11:42.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.323+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5b88003c80 con 0x7f5b98109770 2024-04-22T01:11:42.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.324+0000 7f5b7ffff700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5b800a2eb0 0x7f5b800a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:42.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.325+0000 7f5b9659c700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5b800a2eb0 0x7f5b800a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:42.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.325+0000 7f5b9659c700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5b800a2eb0 0x7f5b800a5370 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b98116950 tx=0x7f5b8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:42.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.326+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b880d0800 con 0x7f5b98109770 2024-04-22T01:11:42.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.326+0000 7f5b7ffff700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] conn(0x7f5b800adf90 0x7f5b800b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:42.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.326+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 --> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b8800ee50 con 0x7f5b800adf90 2024-04-22T01:11:42.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.326+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f5b880d0bf0 con 0x7f5b98109770 2024-04-22T01:11:42.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.330+0000 7f5b9759e700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] conn(0x7f5b800adf90 0x7f5b800b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:42.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.331+0000 7f5b9759e700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] conn(0x7f5b800adf90 0x7f5b800b0430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:42.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.332+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== osd.9 v2:172.21.15.38:6826/1134107438 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b8800ee50 con 0x7f5b800adf90 2024-04-22T01:11:42.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.367+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 --> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b84002de0 con 0x7f5b800adf90 2024-04-22T01:11:42.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.368+0000 7f5b7ffff700 1 -- 172.21.15.38:0/871338797 <== osd.9 v2:172.21.15.38:6826/1134107438 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5b84002de0 con 0x7f5b800adf90 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.368+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] conn(0x7f5b800adf90 msgr2=0x7f5b800b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.368+0000 7f5b9d039700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] conn(0x7f5b800adf90 0x7f5b800b0430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.368+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5b800a2eb0 msgr2=0x7f5b800a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5b800a2eb0 0x7f5b800a5370 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b98116950 tx=0x7f5b8c008040 comp rx=0 tx=0).stop 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98109770 msgr2=0x7f5b9811b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 --2- 172.21.15.38:0/871338797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b98109770 0x7f5b9811b110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5b88006010 tx=0x7f5b8800bfd0 comp rx=0 tx=0).stop 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9759e700 1 -- 172.21.15.38:0/871338797 reap_dead start 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 shutdown_connections 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 >> 172.21.15.38:0/871338797 conn(0x7f5b9806d3a0 msgr2=0x7f5b9811d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:42.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 shutdown_connections 2024-04-22T01:11:42.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:42.369+0000 7f5b9d039700 1 -- 172.21.15.38:0/871338797 wait complete. 2024-04-22T01:11:43.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:43 smithi038 ceph-mon[28043]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:43.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:43 smithi203 ceph-mon[31825]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:43.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:43 smithi111 ceph-mon[31057]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:45.361 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:45.365 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:45.367 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:45.760 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:45 smithi038 ceph-mon[28043]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T01:11:45.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:45 smithi203 ceph-mon[31825]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T01:11:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:45.812 INFO:teuthology.orchestra.run.smithi038.stdout:287762808871 2024-04-22T01:11:45.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-22T01:11:45.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:45 smithi111 ceph-mon[31057]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T01:11:47.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:47 smithi038 ceph-mon[28043]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:47.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:47 smithi203 ceph-mon[31825]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:47 smithi111 ceph-mon[31057]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:49.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 -- 172.21.15.38:0/2616535190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c1024e0 msgr2=0x7fb22c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:49.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 --2- 172.21.15.38:0/2616535190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c1024e0 0x7fb22c1028c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb228009990 tx=0x7fb228009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 -- 172.21.15.38:0/2616535190 shutdown_connections 2024-04-22T01:11:49.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 --2- 172.21.15.38:0/2616535190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c107570 0x7fb22c10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 --2- 172.21.15.38:0/2616535190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c1024e0 0x7fb22c1028c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 --2- 172.21.15.38:0/2616535190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 0x7fb22c106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 -- 172.21.15.38:0/2616535190 >> 172.21.15.38:0/2616535190 conn(0x7fb22c079390 msgr2=0x7fb22c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:49.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 -- 172.21.15.38:0/2616535190 shutdown_connections 2024-04-22T01:11:49.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.288+0000 7fb233592700 1 -- 172.21.15.38:0/2616535190 wait complete. 2024-04-22T01:11:49.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 Processor -- start 2024-04-22T01:11:49.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 -- start start 2024-04-22T01:11:49.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c1024e0 0x7fb22c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 0x7fb22c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 0x7fb22c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c112560 con 0x7fb22c107570 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1123e0 con 0x7fb22c102e90 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb233592700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1126e0 con 0x7fb22c1024e0 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb231b2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 0x7fb22c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb231b2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 0x7fb22c198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46744/0 (socket says 172.21.15.38:46744) 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.289+0000 7fb231b2f700 1 -- 172.21.15.38:0/2689017168 learned_addr learned my addr 172.21.15.38:0/2689017168 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb23132e700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c1024e0 0x7fb22c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb230b2d700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 0x7fb22c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb231b2f700 1 -- 172.21.15.38:0/2689017168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c1024e0 msgr2=0x7fb22c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb231b2f700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c1024e0 0x7fb22c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb231b2f700 1 -- 172.21.15.38:0/2689017168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 msgr2=0x7fb22c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb231b2f700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 0x7fb22c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb231b2f700 1 -- 172.21.15.38:0/2689017168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c198fa0 con 0x7fb22c107570 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb230b2d700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 0x7fb22c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb231b2f700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 0x7fb22c198800 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb224007ee0 tx=0x7fb22400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22400dbd0 con 0x7fb22c107570 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb22400dd70 con 0x7fb22c107570 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c199290 con 0x7fb22c107570 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224005a60 con 0x7fb22c107570 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.290+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c1a54c0 con 0x7fb22c107570 2024-04-22T01:11:49.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.291+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb22c068f20 con 0x7fb22c107570 2024-04-22T01:11:49.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.293+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb22401a020 con 0x7fb22c107570 2024-04-22T01:11:49.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.293+0000 7fb2227fc700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2180a2cb0 0x7fb2180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:49.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.294+0000 7fb23132e700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2180a2cb0 0x7fb2180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:49.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.294+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2240cf240 con 0x7fb22c107570 2024-04-22T01:11:49.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.294+0000 7fb23132e700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2180a2cb0 0x7fb2180a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b5c0 tx=0x7fb228005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:49.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.297+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb224010070 con 0x7fb22c107570 2024-04-22T01:11:49.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.511+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb22c02db30 con 0x7fb22c107570 2024-04-22T01:11:49.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.511+0000 7fb2227fc700 1 -- 172.21.15.38:0/2689017168 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb22409bce0 con 0x7fb22c107570 2024-04-22T01:11:49.512 INFO:teuthology.orchestra.run.smithi038.stdout:141733920834 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2180a2cb0 msgr2=0x7fb2180a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2180a2cb0 0x7fb2180a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b5c0 tx=0x7fb228005960 comp rx=0 tx=0).stop 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 msgr2=0x7fb22c198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 0x7fb22c198800 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb224007ee0 tx=0x7fb22400d3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 shutdown_connections 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb2180a2cb0 0x7fb2180a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb22c107570 0x7fb22c198800 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb22c1024e0 0x7fb22c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 --2- 172.21.15.38:0/2689017168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb22c102e90 0x7fb22c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 >> 172.21.15.38:0/2689017168 conn(0x7fb22c079390 msgr2=0x7fb22c100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:49.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 shutdown_connections 2024-04-22T01:11:49.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:49.513+0000 7fb233592700 1 -- 172.21.15.38:0/2689017168 wait complete. 2024-04-22T01:11:49.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:49 smithi203 ceph-mon[31825]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T01:11:49.827 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:49 smithi038 ceph-mon[28043]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T01:11:49.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:49 smithi111 ceph-mon[31057]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T01:11:50.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:50 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2689017168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T01:11:50.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:50 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2689017168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T01:11:50.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:50 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2689017168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T01:11:51.559 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:51 smithi038 ceph-mon[28043]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:51.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:51 smithi203 ceph-mon[31825]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:51.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:51 smithi111 ceph-mon[31057]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:11:52.050 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:52.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 -- 172.21.15.38:0/1553345087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e41082b0 msgr2=0x7f74e4108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 --2- 172.21.15.38:0/1553345087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e41082b0 0x7f74e4108690 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009990 tx=0x7f74d4009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 -- 172.21.15.38:0/1553345087 shutdown_connections 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 --2- 172.21.15.38:0/1553345087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e406a8e0 0x7f74e410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 --2- 172.21.15.38:0/1553345087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e41082b0 0x7f74e4108690 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 --2- 172.21.15.38:0/1553345087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e4069f40 0x7f74e406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 -- 172.21.15.38:0/1553345087 >> 172.21.15.38:0/1553345087 conn(0x7f74e40780e0 msgr2=0x7f74e40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 -- 172.21.15.38:0/1553345087 shutdown_connections 2024-04-22T01:11:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.087+0000 7f74ea34f700 1 -- 172.21.15.38:0/1553345087 wait complete. 2024-04-22T01:11:52.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 Processor -- start 2024-04-22T01:11:52.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 -- start start 2024-04-22T01:11:52.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e4069f40 0x7f74e419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:52.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 0x7f74e419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e41082b0 0x7f74e4198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4112510 con 0x7f74e406a8e0 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4112390 con 0x7f74e4069f40 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74ea34f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4112690 con 0x7f74e41082b0 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 0x7f74e419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.088+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 0x7f74e419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46766/0 (socket says 172.21.15.38:46766) 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 -- 172.21.15.38:0/3025965493 learned_addr learned my addr 172.21.15.38:0/3025965493 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e3fff700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e4069f40 0x7f74e419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e88ec700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e41082b0 0x7f74e4198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 -- 172.21.15.38:0/3025965493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e41082b0 msgr2=0x7f74e4198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e41082b0 0x7f74e4198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 -- 172.21.15.38:0/3025965493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e4069f40 msgr2=0x7f74e419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e4069f40 0x7f74e419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 -- 172.21.15.38:0/3025965493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e4199030 con 0x7f74e406a8e0 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e88ec700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e41082b0 0x7f74e4198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e3fff700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e4069f40 0x7f74e419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e37fe700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 0x7f74e419ec10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f74d800c970 tx=0x7f74d800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8007aa0 con 0x7f74e406a8e0 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d8004d10 con 0x7f74e406a8e0 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e4199320 con 0x7f74e406a8e0 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8005690 con 0x7f74e406a8e0 2024-04-22T01:11:52.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.089+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e41a5530 con 0x7f74e406a8e0 2024-04-22T01:11:52.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.090+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e4103960 con 0x7f74e406a8e0 2024-04-22T01:11:52.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.092+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f74d8004750 con 0x7f74e406a8e0 2024-04-22T01:11:52.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.092+0000 7f74e17fa700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74cc0a2d30 0x7f74cc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:52.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.092+0000 7f74e3fff700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74cc0a2d30 0x7f74cc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:52.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.092+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74d80cf290 con 0x7f74e406a8e0 2024-04-22T01:11:52.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.093+0000 7f74e3fff700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74cc0a2d30 0x7f74cc0a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006010 tx=0x7f74d400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:52.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.098+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f74d809bdc0 con 0x7f74e406a8e0 2024-04-22T01:11:52.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.305+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f74e402db30 con 0x7f74e406a8e0 2024-04-22T01:11:52.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.306+0000 7f74e17fa700 1 -- 172.21.15.38:0/3025965493 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f74d809b6c0 con 0x7f74e406a8e0 2024-04-22T01:11:52.306 INFO:teuthology.orchestra.run.smithi038.stdout:219043332146 2024-04-22T01:11:52.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74cc0a2d30 msgr2=0x7f74cc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:52.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74cc0a2d30 0x7f74cc0a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006010 tx=0x7f74d400b540 comp rx=0 tx=0).stop 2024-04-22T01:11:52.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 msgr2=0x7f74e419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:52.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 0x7f74e419ec10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f74d800c970 tx=0x7f74d800cd30 comp rx=0 tx=0).stop 2024-04-22T01:11:52.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 shutdown_connections 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74cc0a2d30 0x7f74cc0a51f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74e41082b0 0x7f74e4198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74e4069f40 0x7f74e419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 --2- 172.21.15.38:0/3025965493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e406a8e0 0x7f74e419ec10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 >> 172.21.15.38:0/3025965493 conn(0x7f74e40780e0 msgr2=0x7f74e40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 shutdown_connections 2024-04-22T01:11:52.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:52.308+0000 7f74ea34f700 1 -- 172.21.15.38:0/3025965493 wait complete. 2024-04-22T01:11:52.633 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:52 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3025965493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T01:11:52.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:52 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3025965493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T01:11:52.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:52 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3025965493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T01:11:53.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:53 smithi203 ceph-mon[31825]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:53.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:53 smithi038 ceph-mon[28043]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:53 smithi111 ceph-mon[31057]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:11:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 -- 172.21.15.38:0/748756970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 msgr2=0x7f9774108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 --2- 172.21.15.38:0/748756970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774108690 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9770009990 tx=0x7f9770009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 -- 172.21.15.38:0/748756970 shutdown_connections 2024-04-22T01:11:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 --2- 172.21.15.38:0/748756970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 0x7f977410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 --2- 172.21.15.38:0/748756970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774108690 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 --2- 172.21.15.38:0/748756970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9774069f40 0x7f977406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 -- 172.21.15.38:0/748756970 >> 172.21.15.38:0/748756970 conn(0x7f97740780e0 msgr2=0x7f97740784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 -- 172.21.15.38:0/748756970 shutdown_connections 2024-04-22T01:11:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.197+0000 7f977c074700 1 -- 172.21.15.38:0/748756970 wait complete. 2024-04-22T01:11:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 Processor -- start 2024-04-22T01:11:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 -- start start 2024-04-22T01:11:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9774069f40 0x7f977419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 0x7f977419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9774112510 con 0x7f97741082b0 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9774112390 con 0x7f9774069f40 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977c074700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9774112690 con 0x7f977406a8e0 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977a611700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977a611700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46780/0 (socket says 172.21.15.38:46780) 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977a611700 1 -- 172.21.15.38:0/2686433936 learned_addr learned my addr 172.21.15.38:0/2686433936 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f9779e10700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9774069f40 0x7f977419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.198+0000 7f977960f700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 0x7f977419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977a611700 1 -- 172.21.15.38:0/2686433936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 msgr2=0x7f977419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977a611700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 0x7f977419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977a611700 1 -- 172.21.15.38:0/2686433936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9774069f40 msgr2=0x7f977419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977a611700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9774069f40 0x7f977419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977a611700 1 -- 172.21.15.38:0/2686433936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9774198ff0 con 0x7f97741082b0 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977960f700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 0x7f977419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977a611700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774198890 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f976c00ada0 tx=0x7f976c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c017020 con 0x7f97741082b0 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976c00a0d0 con 0x7f97741082b0 2024-04-22T01:11:54.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97741992e0 con 0x7f97741082b0 2024-04-22T01:11:54.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c005480 con 0x7f97741082b0 2024-04-22T01:11:54.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.199+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97741a55a0 con 0x7f97741082b0 2024-04-22T01:11:54.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.200+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9774068f20 con 0x7f97741082b0 2024-04-22T01:11:54.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.201+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f976c005620 con 0x7f97741082b0 2024-04-22T01:11:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.202+0000 7f976affd700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f97600a2d30 0x7f97600a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.202+0000 7f9779e10700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f97600a2d30 0x7f97600a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.202+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f976c0ceb70 con 0x7f97741082b0 2024-04-22T01:11:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.203+0000 7f9779e10700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f97600a2d30 0x7f97600a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f97741999d0 tx=0x7f9770005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:54.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.207+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f976c021020 con 0x7f97741082b0 2024-04-22T01:11:54.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.417+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9774199dc0 con 0x7f97741082b0 2024-04-22T01:11:54.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.417+0000 7f976affd700 1 -- 172.21.15.38:0/2686433936 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f976c09b6a0 con 0x7f97741082b0 2024-04-22T01:11:54.417 INFO:teuthology.orchestra.run.smithi038.stdout:34359738454 2024-04-22T01:11:54.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f97600a2d30 msgr2=0x7f97600a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f97600a2d30 0x7f97600a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f97741999d0 tx=0x7f9770005960 comp rx=0 tx=0).stop 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 msgr2=0x7f9774198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774198890 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f976c00ada0 tx=0x7f976c00c330 comp rx=0 tx=0).stop 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 shutdown_connections 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f97600a2d30 0x7f97600a51f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97741082b0 0x7f9774198890 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9774069f40 0x7f977419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 --2- 172.21.15.38:0/2686433936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f977406a8e0 0x7f977419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 >> 172.21.15.38:0/2686433936 conn(0x7f97740780e0 msgr2=0x7f97740fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 shutdown_connections 2024-04-22T01:11:54.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:54.419+0000 7f977c074700 1 -- 172.21.15.38:0/2686433936 wait complete. 2024-04-22T01:11:54.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2686433936' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T01:11:54.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2686433936' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T01:11:54.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2686433936' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T01:11:55.077 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332141 got 219043332146 for osd.7 2024-04-22T01:11:55.078 DEBUG:teuthology.parallel:result is None 2024-04-22T01:11:55.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:55 smithi038 ceph-mon[28043]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:11:55.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:55 smithi203 ceph-mon[31825]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 -- 172.21.15.38:0/2318554656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc102e90 msgr2=0x7fe6bc106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/2318554656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc102e90 0x7fe6bc106e90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009990 tx=0x7fe6b0009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 -- 172.21.15.38:0/2318554656 shutdown_connections 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/2318554656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6bc107570 0x7fe6bc10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/2318554656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6bc1024e0 0x7fe6bc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/2318554656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc102e90 0x7fe6bc106e90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 -- 172.21.15.38:0/2318554656 >> 172.21.15.38:0/2318554656 conn(0x7fe6bc079390 msgr2=0x7fe6bc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 -- 172.21.15.38:0/2318554656 shutdown_connections 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.768+0000 7fe6c22cc700 1 -- 172.21.15.38:0/2318554656 wait complete. 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 Processor -- start 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 -- start start 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6bc1024e0 0x7fe6bc19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6bc102e90 0x7fe6bc19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 0x7fe6bc198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc112560 con 0x7fe6bc107570 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1123e0 con 0x7fe6bc102e90 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c22cc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1126e0 con 0x7fe6bc1024e0 2024-04-22T01:11:55.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c0869700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 0x7fe6bc198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6bbfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6bc1024e0 0x7fe6bc19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.769+0000 7fe6c0869700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 0x7fe6bc198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46814/0 (socket says 172.21.15.38:46814) 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 -- 172.21.15.38:0/1303218634 learned_addr learned my addr 172.21.15.38:0/1303218634 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6bb7fe700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6bc102e90 0x7fe6bc19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 -- 172.21.15.38:0/1303218634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6bc1024e0 msgr2=0x7fe6bc19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6bc1024e0 0x7fe6bc19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 -- 172.21.15.38:0/1303218634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6bc102e90 msgr2=0x7fe6bc19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6bc102e90 0x7fe6bc19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 -- 172.21.15.38:0/1303218634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc198fe0 con 0x7fe6bc107570 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c0869700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 0x7fe6bc198880 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4007ee0 tx=0x7fe6b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4004bb0 con 0x7fe6bc107570 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc1992d0 con 0x7fe6bc107570 2024-04-22T01:11:55.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b4004d50 con 0x7fe6bc107570 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4005960 con 0x7fe6bc107570 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.770+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc1a5590 con 0x7fe6bc107570 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.771+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6bc068f20 con 0x7fe6bc107570 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.772+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe6b401a020 con 0x7fe6bc107570 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.773+0000 7fe6b97fa700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6a40a2cb0 0x7fe6a40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.773+0000 7fe6bbfff700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6a40a2cb0 0x7fe6a40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.773+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe6b40cf250 con 0x7fe6bc107570 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.773+0000 7fe6bbfff700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6a40a2cb0 0x7fe6a40a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc1999c0 tx=0x7fe6ac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:55.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.777+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe6b4010070 con 0x7fe6bc107570 2024-04-22T01:11:55.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:55 smithi111 ceph-mon[31057]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:11:55.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.987+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fe6bc02db30 con 0x7fe6bc107570 2024-04-22T01:11:55.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.988+0000 7fe6b97fa700 1 -- 172.21.15.38:0/1303218634 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe6b409bcf0 con 0x7fe6bc107570 2024-04-22T01:11:55.988 INFO:teuthology.orchestra.run.smithi038.stdout:253403070511 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6a40a2cb0 msgr2=0x7fe6a40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6a40a2cb0 0x7fe6a40a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc1999c0 tx=0x7fe6ac00b410 comp rx=0 tx=0).stop 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 msgr2=0x7fe6bc198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 0x7fe6bc198880 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4007ee0 tx=0x7fe6b400d3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 shutdown_connections 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6a40a2cb0 0x7fe6a40a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6bc107570 0x7fe6bc198880 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6bc1024e0 0x7fe6bc19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 --2- 172.21.15.38:0/1303218634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6bc102e90 0x7fe6bc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 >> 172.21.15.38:0/1303218634 conn(0x7fe6bc079390 msgr2=0x7fe6bc101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 shutdown_connections 2024-04-22T01:11:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.990+0000 7fe6c22cc700 1 -- 172.21.15.38:0/1303218634 wait complete. 2024-04-22T01:11:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1154045071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T01:11:56.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1303218634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T01:11:56.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1154045071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T01:11:56.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1303218634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T01:11:56.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1154045071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T01:11:56.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1303218634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T01:11:57.206 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920834 for osd.4 2024-04-22T01:11:57.207 DEBUG:teuthology.parallel:result is None 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 -- 172.21.15.38:0/4131335684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101130 msgr2=0x7f1d0c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/4131335684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101130 0x7f1d0c1015b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1d00009990 tx=0x7f1d00009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 -- 172.21.15.38:0/4131335684 shutdown_connections 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/4131335684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d0c101af0 0x7f1d0c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/4131335684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d0c1071a0 0x7f1d0c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/4131335684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101130 0x7f1d0c1015b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 -- 172.21.15.38:0/4131335684 >> 172.21.15.38:0/4131335684 conn(0x7f1d0c0780e0 msgr2=0x7f1d0c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 -- 172.21.15.38:0/4131335684 shutdown_connections 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.274+0000 7f1d12fe7700 1 -- 172.21.15.38:0/4131335684 wait complete. 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 Processor -- start 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 -- start start 2024-04-22T01:11:57.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d0c101130 0x7f1d0c199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 0x7f1d0c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d0c1071a0 0x7f1d0c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c111430 con 0x7f1d0c101af0 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1112b0 con 0x7f1d0c101130 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d12fe7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1115b0 con 0x7f1d0c1071a0 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d0bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 0x7f1d0c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d0bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 0x7f1d0c199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46800/0 (socket says 172.21.15.38:46800) 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d0bfff700 1 -- 172.21.15.38:0/1154045071 learned_addr learned my addr 172.21.15.38:0/1154045071 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d10d83700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d0c101130 0x7f1d0c199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.275+0000 7f1d11584700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d0c1071a0 0x7f1d0c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d0bfff700 1 -- 172.21.15.38:0/1154045071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d0c1071a0 msgr2=0x7f1d0c19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d0bfff700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d0c1071a0 0x7f1d0c19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d0bfff700 1 -- 172.21.15.38:0/1154045071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d0c101130 msgr2=0x7f1d0c199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:57.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d0bfff700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d0c101130 0x7f1d0c199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d0bfff700 1 -- 172.21.15.38:0/1154045071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0c1a4d80 con 0x7f1d0c101af0 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stdout:197568495673 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d0bfff700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 0x7f1d0c199ed0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1d00000c00 tx=0x7f1d00003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00027070 con 0x7f1d0c101af0 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d00004190 con 0x7f1d0c101af0 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0c1a5010 con 0x7f1d0c101af0 2024-04-22T01:11:57.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0000f460 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.276+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0c1a54c0 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.277+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d0c068f20 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.278+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1d0002a070 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.279+0000 7f1d09ffb700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1cf40a2d30 0x7f1cf40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.279+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1d000d0800 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.279+0000 7f1d10d83700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1cf40a2d30 0x7f1cf40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.279+0000 7f1d10d83700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1cf40a2d30 0x7f1cf40a51f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005fd0 tx=0x7f1cfc005900 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.282+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1d00013070 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.492+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1d0c02db30 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.492+0000 7f1d09ffb700 1 -- 172.21.15.38:0/1154045071 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1d0009d330 con 0x7f1d0c101af0 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1cf40a2d30 msgr2=0x7f1cf40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1cf40a2d30 0x7f1cf40a51f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005fd0 tx=0x7f1cfc005900 comp rx=0 tx=0).stop 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 msgr2=0x7f1d0c199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 0x7f1d0c199ed0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1d00000c00 tx=0x7f1d00003730 comp rx=0 tx=0).stop 2024-04-22T01:11:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 shutdown_connections 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1cf40a2d30 0x7f1cf40a51f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d0c1071a0 0x7f1d0c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d0c101130 0x7f1d0c199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 --2- 172.21.15.38:0/1154045071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c101af0 0x7f1d0c199ed0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 >> 172.21.15.38:0/1154045071 conn(0x7f1d0c0780e0 msgr2=0x7f1d0c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 shutdown_connections 2024-04-22T01:11:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:55.495+0000 7f1d12fe7700 1 -- 172.21.15.38:0/1154045071 wait complete. 2024-04-22T01:11:57.510 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:11:57.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:57 smithi203 ceph-mon[31825]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T01:11:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:57 smithi111 ceph-mon[31057]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T01:11:57.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:57 smithi038 ceph-mon[28043]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T01:11:57.937 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495673 for osd.6 2024-04-22T01:11:57.937 DEBUG:teuthology.parallel:result is None 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 -- 172.21.15.38:0/3262819514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 msgr2=0x7fce701086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 --2- 172.21.15.38:0/3262819514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701086a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fce6800b0a0 tx=0x7fce6800b3b0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 -- 172.21.15.38:0/3262819514 shutdown_connections 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 --2- 172.21.15.38:0/3262819514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce701011c0 0x7fce7010f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 --2- 172.21.15.38:0/3262819514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701086a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 --2- 172.21.15.38:0/3262819514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce70100820 0x7fce70100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 -- 172.21.15.38:0/3262819514 >> 172.21.15.38:0/3262819514 conn(0x7fce700777a0 msgr2=0x7fce700feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:58.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 -- 172.21.15.38:0/3262819514 shutdown_connections 2024-04-22T01:11:58.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.125+0000 7fce76e99700 1 -- 172.21.15.38:0/3262819514 wait complete. 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 Processor -- start 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 -- start start 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce70100820 0x7fce7019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce701011c0 0x7fce7019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fce70112520 con 0x7fce701082c0 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fce701123a0 con 0x7fce70100820 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce76e99700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fce701126a0 con 0x7fce701011c0 2024-04-22T01:11:58.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce75436700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce75436700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42186/0 (socket says 172.21.15.38:42186) 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce75436700 1 -- 172.21.15.38:0/1206321496 learned_addr learned my addr 172.21.15.38:0/1206321496 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce74c35700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce70100820 0x7fce7019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.126+0000 7fce6ffff700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce701011c0 0x7fce7019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce75436700 1 -- 172.21.15.38:0/1206321496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce701011c0 msgr2=0x7fce7019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce75436700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce701011c0 0x7fce7019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce75436700 1 -- 172.21.15.38:0/1206321496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce70100820 msgr2=0x7fce7019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce75436700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce70100820 0x7fce7019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce75436700 1 -- 172.21.15.38:0/1206321496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce70198fa0 con 0x7fce701082c0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce75436700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701987c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce6000d980 tx=0x7fce6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60017070 con 0x7fce701082c0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce70199290 con 0x7fce701082c0 2024-04-22T01:11:58.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce60005020 con 0x7fce701082c0 2024-04-22T01:11:58.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6000fbf0 con 0x7fce701082c0 2024-04-22T01:11:58.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.127+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce701a54a0 con 0x7fce701082c0 2024-04-22T01:11:58.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.128+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce70103b40 con 0x7fce701082c0 2024-04-22T01:11:58.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.129+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fce600051d0 con 0x7fce701082c0 2024-04-22T01:11:58.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.130+0000 7fce6dffb700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce580a2cb0 0x7fce580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:58.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.130+0000 7fce74c35700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce580a2cb0 0x7fce580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:58.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.130+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fce600cf220 con 0x7fce701082c0 2024-04-22T01:11:58.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.131+0000 7fce74c35700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce580a2cb0 0x7fce580a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fce68009e60 tx=0x7fce6800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:58.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.135+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fce60020020 con 0x7fce701082c0 2024-04-22T01:11:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.345+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fce7002db30 con 0x7fce701082c0 2024-04-22T01:11:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.345+0000 7fce6dffb700 1 -- 172.21.15.38:0/1206321496 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fce6009bcc0 con 0x7fce701082c0 2024-04-22T01:11:58.346 INFO:teuthology.orchestra.run.smithi038.stdout:343597383711 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce580a2cb0 msgr2=0x7fce580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce580a2cb0 0x7fce580a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fce68009e60 tx=0x7fce6800b7b0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 msgr2=0x7fce701987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701987c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce6000d980 tx=0x7fce6000dc90 comp rx=0 tx=0).stop 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 shutdown_connections 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce580a2cb0 0x7fce580a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce701082c0 0x7fce701987c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce70100820 0x7fce7019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 --2- 172.21.15.38:0/1206321496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce701011c0 0x7fce7019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 >> 172.21.15.38:0/1206321496 conn(0x7fce700777a0 msgr2=0x7fce700feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 shutdown_connections 2024-04-22T01:11:58.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:58.347+0000 7fce76e99700 1 -- 172.21.15.38:0/1206321496 wait complete. 2024-04-22T01:11:58.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:58 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1206321496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T01:11:58.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:58 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1206321496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T01:11:58.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:58 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1206321496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T01:11:59.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383707 got 343597383711 for osd.11 2024-04-22T01:11:59.161 DEBUG:teuthology.parallel:result is None 2024-04-22T01:11:59.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:11:59 smithi038 ceph-mon[28043]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:11:59.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:11:59 smithi203 ceph-mon[31825]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:11:59.770 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070511 for osd.8 2024-04-22T01:11:59.770 DEBUG:teuthology.parallel:result is None 2024-04-22T01:11:59.796 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:11:59 smithi111 ceph-mon[31057]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 -- 172.21.15.38:0/1370591271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc1071a0 msgr2=0x7f7efc107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 --2- 172.21.15.38:0/1370591271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc1071a0 0x7f7efc107580 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009990 tx=0x7f7ef8009ca0 comp rx=0 tx=0).stop 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 -- 172.21.15.38:0/1370591271 shutdown_connections 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 --2- 172.21.15.38:0/1370591271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101af0 0x7f7efc10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 --2- 172.21.15.38:0/1370591271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc1071a0 0x7f7efc107580 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 --2- 172.21.15.38:0/1370591271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7efc101130 0x7f7efc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.962+0000 7f7f034c6700 1 -- 172.21.15.38:0/1370591271 >> 172.21.15.38:0/1370591271 conn(0x7f7efc0780e0 msgr2=0x7f7efc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:11:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.963+0000 7f7f034c6700 1 -- 172.21.15.38:0/1370591271 shutdown_connections 2024-04-22T01:11:59.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.963+0000 7f7f034c6700 1 -- 172.21.15.38:0/1370591271 wait complete. 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.963+0000 7f7f034c6700 1 Processor -- start 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.963+0000 7f7f034c6700 1 -- start start 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f034c6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 0x7f7efc102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f034c6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc101af0 0x7f7efc103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f034c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7efc1071a0 0x7f7efc103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f034c6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc111410 con 0x7f7efc101af0 2024-04-22T01:11:59.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f034c6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc111290 con 0x7f7efc101130 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f034c6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc111590 con 0x7f7efc1071a0 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 0x7f7efc102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f00a61700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc101af0 0x7f7efc103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 0x7f7efc102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:49898/0 (socket says 172.21.15.38:49898) 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 -- 172.21.15.38:0/2407774136 learned_addr learned my addr 172.21.15.38:0/2407774136 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01a63700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7efc1071a0 0x7f7efc103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 -- 172.21.15.38:0/2407774136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7efc1071a0 msgr2=0x7f7efc103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7efc1071a0 0x7f7efc103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 -- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc101af0 msgr2=0x7f7efc103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc101af0 0x7f7efc103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 -- 172.21.15.38:0/2407774136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7efc1ab340 con 0x7f7efc101130 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f00a61700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc101af0 0x7f7efc103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.964+0000 7f7f01262700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 0x7f7efc102e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8000c00 tx=0x7f7ef800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:59.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.965+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8027070 con 0x7f7efc101130 2024-04-22T01:11:59.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.965+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ef8003d70 con 0x7f7efc101130 2024-04-22T01:11:59.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.965+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef800f460 con 0x7f7efc101130 2024-04-22T01:11:59.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.965+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7efc1ab570 con 0x7f7efc101130 2024-04-22T01:11:59.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.965+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7efc1ab910 con 0x7f7efc101130 2024-04-22T01:11:59.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.966+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7efc102850 con 0x7f7efc101130 2024-04-22T01:11:59.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.967+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7ef802a070 con 0x7f7efc101130 2024-04-22T01:11:59.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.968+0000 7f7ef27fc700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7ee80a2cb0 0x7f7ee80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:11:59.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.968+0000 7f7f00a61700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7ee80a2cb0 0x7f7ee80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:11:59.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.968+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ef80d02d0 con 0x7f7efc101130 2024-04-22T01:11:59.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.969+0000 7f7f00a61700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7ee80a2cb0 0x7f7ee80a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7efc1045d0 tx=0x7f7eec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:11:59.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:11:59.973+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7ef8013070 con 0x7f7efc101130 2024-04-22T01:12:00.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.182+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7efc1abbd0 con 0x7f7efc101130 2024-04-22T01:12:00.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.182+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2407774136 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7ef809ce00 con 0x7f7efc101130 2024-04-22T01:12:00.183 INFO:teuthology.orchestra.run.smithi038.stdout:77309411406 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7ee80a2cb0 msgr2=0x7f7ee80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7ee80a2cb0 0x7f7ee80a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7efc1045d0 tx=0x7f7eec00b410 comp rx=0 tx=0).stop 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 msgr2=0x7f7efc102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 0x7f7efc102e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8000c00 tx=0x7f7ef800bab0 comp rx=0 tx=0).stop 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 shutdown_connections 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7ee80a2cb0 0x7f7ee80a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7efc1071a0 0x7f7efc103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7efc101130 0x7f7efc102e10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 --2- 172.21.15.38:0/2407774136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc101af0 0x7f7efc103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:00.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.184+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 >> 172.21.15.38:0/2407774136 conn(0x7f7efc0780e0 msgr2=0x7f7efc1003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.185+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 shutdown_connections 2024-04-22T01:12:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:00.185+0000 7f7f034c6700 1 -- 172.21.15.38:0/2407774136 wait complete. 2024-04-22T01:12:00.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:00 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2407774136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T01:12:00.794 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:00 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2407774136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T01:12:00.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:00 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2407774136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T01:12:01.274 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:01.275 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738454 for osd.0 2024-04-22T01:12:01.275 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:01.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:01 smithi203 ceph-mon[31825]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:01.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:01 smithi111 ceph-mon[31057]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:01.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:01 smithi038 ceph-mon[28043]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:02.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411406 for osd.2 2024-04-22T01:12:02.280 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 -- 172.21.15.38:0/990691134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 msgr2=0x7fb4dc10df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/990691134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc10df70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400b0d0 tx=0x7fb4d400b3e0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 -- 172.21.15.38:0/990691134 shutdown_connections 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/990691134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc10df70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/990691134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc1012c0 0x7fb4dc1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/990691134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4dc101c70 0x7fb4dc105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 -- 172.21.15.38:0/990691134 >> 172.21.15.38:0/990691134 conn(0x7fb4dc0fcb50 msgr2=0x7fb4dc0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:02.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 -- 172.21.15.38:0/990691134 shutdown_connections 2024-04-22T01:12:02.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.459+0000 7fb4e0a03700 1 -- 172.21.15.38:0/990691134 wait complete. 2024-04-22T01:12:02.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.460+0000 7fb4e0a03700 1 Processor -- start 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 -- start start 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc1012c0 0x7fb4dc19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4dc101c70 0x7fb4dc19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc1113e0 con 0x7fb4dc1063a0 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc111260 con 0x7fb4dc101c70 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4e0a03700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4dc111560 con 0x7fb4dc1012c0 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42210/0 (socket says 172.21.15.38:42210) 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 -- 172.21.15.38:0/2031046267 learned_addr learned my addr 172.21.15.38:0/2031046267 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4da59c700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4dc101c70 0x7fb4dc19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4dad9d700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc1012c0 0x7fb4dc19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 -- 172.21.15.38:0/2031046267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc1012c0 msgr2=0x7fb4dc19e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc1012c0 0x7fb4dc19e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 -- 172.21.15.38:0/2031046267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4dc101c70 msgr2=0x7fb4dc19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4dc101c70 0x7fb4dc19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.461+0000 7fb4db59e700 1 -- 172.21.15.38:0/2031046267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4dc198f50 con 0x7fb4dc1063a0 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.462+0000 7fb4db59e700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc198670 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4004dc0 tx=0x7fb4d4004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.462+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4021020 con 0x7fb4dc1063a0 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.462+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4d4007c00 con 0x7fb4dc1063a0 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.462+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4dc1991e0 con 0x7fb4dc1063a0 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.462+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4005370 con 0x7fb4dc1063a0 2024-04-22T01:12:02.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.462+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4dc1a5520 con 0x7fb4dc1063a0 2024-04-22T01:12:02.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.463+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4dc199550 con 0x7fb4dc1063a0 2024-04-22T01:12:02.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.464+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb4d4024020 con 0x7fb4dc1063a0 2024-04-22T01:12:02.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.465+0000 7fb4d3fff700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb4c80a2cb0 0x7fb4c80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:02.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.465+0000 7fb4dad9d700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb4c80a2cb0 0x7fb4c80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:02.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.465+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb4d400f020 con 0x7fb4dc1063a0 2024-04-22T01:12:02.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.465+0000 7fb4dad9d700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb4c80a2cb0 0x7fb4c80a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc005ea0 tx=0x7fb4cc005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:02.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.470+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb4dc199550 con 0x7fb4dc1063a0 2024-04-22T01:12:02.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.684+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb4dc199ba0 con 0x7fb4dc1063a0 2024-04-22T01:12:02.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.685+0000 7fb4d3fff700 1 -- 172.21.15.38:0/2031046267 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb4d4007db0 con 0x7fb4dc1063a0 2024-04-22T01:12:02.685 INFO:teuthology.orchestra.run.smithi038.stdout:163208757311 2024-04-22T01:12:02.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb4c80a2cb0 msgr2=0x7fb4c80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:02.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb4c80a2cb0 0x7fb4c80a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc005ea0 tx=0x7fb4cc005e30 comp rx=0 tx=0).stop 2024-04-22T01:12:02.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 msgr2=0x7fb4dc198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc198670 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4004dc0 tx=0x7fb4d4004df0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 shutdown_connections 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb4c80a2cb0 0x7fb4c80a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4dc1063a0 0x7fb4dc198670 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4dc1012c0 0x7fb4dc19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 --2- 172.21.15.38:0/2031046267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4dc101c70 0x7fb4dc19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 >> 172.21.15.38:0/2031046267 conn(0x7fb4dc0fcb50 msgr2=0x7fb4dc104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 shutdown_connections 2024-04-22T01:12:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:02.687+0000 7fb4e0a03700 1 -- 172.21.15.38:0/2031046267 wait complete. 2024-04-22T01:12:03.547 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757311 for osd.5 2024-04-22T01:12:03.548 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:03.779 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:03 smithi038 ceph-mon[28043]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:12:03.779 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:03 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2031046267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T01:12:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:03 smithi111 ceph-mon[31057]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:12:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:03 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2031046267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T01:12:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:03 smithi203 ceph-mon[31825]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:12:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:03 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2031046267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T01:12:05.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:05 smithi038 ceph-mon[28043]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:05 smithi111 ceph-mon[31057]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:05.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 -- 172.21.15.38:0/1134765042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d801082b0 msgr2=0x7f3d80108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:05.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 --2- 172.21.15.38:0/1134765042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d801082b0 0x7f3d80108690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009990 tx=0x7f3d70009ca0 comp rx=0 tx=0).stop 2024-04-22T01:12:05.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 -- 172.21.15.38:0/1134765042 shutdown_connections 2024-04-22T01:12:05.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 --2- 172.21.15.38:0/1134765042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 0x7f3d8010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 --2- 172.21.15.38:0/1134765042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d801082b0 0x7f3d80108690 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 --2- 172.21.15.38:0/1134765042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d80069f40 0x7f3d8006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 -- 172.21.15.38:0/1134765042 >> 172.21.15.38:0/1134765042 conn(0x7f3d800780e0 msgr2=0x7f3d800784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 -- 172.21.15.38:0/1134765042 shutdown_connections 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.983+0000 7f3d84b29700 1 -- 172.21.15.38:0/1134765042 wait complete. 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.984+0000 7f3d84b29700 1 Processor -- start 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.984+0000 7f3d84b29700 1 -- start start 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d84b29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 0x7f3d8019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d84b29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 0x7f3d8019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d84b29700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801082b0 0x7f3d80198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d84b29700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3d801124f0 con 0x7f3d80069f40 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d84b29700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3d80112370 con 0x7f3d8006a8e0 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d84b29700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d80112670 con 0x7f3d801082b0 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 0x7f3d8019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:05.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 0x7f3d8019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42236/0 (socket says 172.21.15.38:42236) 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 -- 172.21.15.38:0/3050425563 learned_addr learned my addr 172.21.15.38:0/3050425563 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7ed9d700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801082b0 0x7f3d80198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7dd9b700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 0x7f3d8019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 -- 172.21.15.38:0/3050425563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801082b0 msgr2=0x7f3d80198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801082b0 0x7f3d80198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 -- 172.21.15.38:0/3050425563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 msgr2=0x7f3d8019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 0x7f3d8019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7e59c700 1 -- 172.21.15.38:0/3050425563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d80199010 con 0x7f3d80069f40 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.985+0000 7f3d7dd9b700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 0x7f3d8019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.986+0000 7f3d7e59c700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 0x7f3d8019e6b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000c00 tx=0x7f3d700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.986+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70027070 con 0x7f3d80069f40 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.986+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d801992a0 con 0x7f3d80069f40 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.986+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d70003780 con 0x7f3d80069f40 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.986+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70022dd0 con 0x7f3d80069f40 2024-04-22T01:12:05.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.986+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d801a5510 con 0x7f3d80069f40 2024-04-22T01:12:05.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.987+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d80103960 con 0x7f3d80069f40 2024-04-22T01:12:05.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.988+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3d7002a070 con 0x7f3d80069f40 2024-04-22T01:12:05.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.989+0000 7f3d6f7fe700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3d680a2d30 0x7f3d680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:05.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.989+0000 7f3d7dd9b700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3d680a2d30 0x7f3d680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:05.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.989+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3d700d0180 con 0x7f3d80069f40 2024-04-22T01:12:05.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.990+0000 7f3d7dd9b700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3d680a2d30 0x7f3d680a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3d74005950 tx=0x7f3d740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:05.998+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3d70013070 con 0x7f3d80069f40 2024-04-22T01:12:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:05 smithi203 ceph-mon[31825]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:06.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.216+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3d8002db30 con 0x7f3d80069f40 2024-04-22T01:12:06.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.217+0000 7f3d6f7fe700 1 -- 172.21.15.38:0/3050425563 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3d70098030 con 0x7f3d80069f40 2024-04-22T01:12:06.217 INFO:teuthology.orchestra.run.smithi038.stdout:107374182474 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3d680a2d30 msgr2=0x7f3d680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3d680a2d30 0x7f3d680a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3d74005950 tx=0x7f3d740058e0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 msgr2=0x7f3d8019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 0x7f3d8019e6b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000c00 tx=0x7f3d700047b0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 shutdown_connections 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3d680a2d30 0x7f3d680a51f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d801082b0 0x7f3d80198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d80069f40 0x7f3d8019e6b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 --2- 172.21.15.38:0/3050425563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d8006a8e0 0x7f3d8019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.220+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 >> 172.21.15.38:0/3050425563 conn(0x7f3d800780e0 msgr2=0x7f3d800fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.221+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 shutdown_connections 2024-04-22T01:12:06.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.221+0000 7f3d84b29700 1 -- 172.21.15.38:0/3050425563 wait complete. 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 -- 172.21.15.38:0/3773493393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4107670 msgr2=0x7f11a410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 --2- 172.21.15.38:0/3773493393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4107670 0x7f11a410f1d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f119c00b0d0 tx=0x7f119c00b3e0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 -- 172.21.15.38:0/3773493393 shutdown_connections 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 --2- 172.21.15.38:0/3773493393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4107670 0x7f11a410f1d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 --2- 172.21.15.38:0/3773493393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a4100e50 0x7f11a4101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 --2- 172.21.15.38:0/3773493393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a4101800 0x7f11a4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 -- 172.21.15.38:0/3773493393 >> 172.21.15.38:0/3773493393 conn(0x7f11a4076450 msgr2=0x7f11a4076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:06.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 -- 172.21.15.38:0/3773493393 shutdown_connections 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.635+0000 7f11a95be700 1 -- 172.21.15.38:0/3773493393 wait complete. 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 Processor -- start 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 -- start start 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a4100e50 0x7f11a419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 0x7f11a419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a4107670 0x7f11a4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f11a41125e0 con 0x7f11a4101800 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f11a4112460 con 0x7f11a4107670 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a95be700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f11a4112760 con 0x7f11a4100e50 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a27fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 0x7f11a419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a27fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 0x7f11a419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42250/0 (socket says 172.21.15.38:42250) 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a27fc700 1 -- 172.21.15.38:0/1566281993 learned_addr learned my addr 172.21.15.38:0/1566281993 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a2ffd700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a4100e50 0x7f11a419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.636+0000 7f11a37fe700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a4107670 0x7f11a4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a27fc700 1 -- 172.21.15.38:0/1566281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a4100e50 msgr2=0x7f11a419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a27fc700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a4100e50 0x7f11a419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a27fc700 1 -- 172.21.15.38:0/1566281993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a4107670 msgr2=0x7f11a4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a27fc700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a4107670 0x7f11a4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a27fc700 1 -- 172.21.15.38:0/1566281993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a4199040 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a27fc700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 0x7f11a419ebd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f119800b530 tx=0x7f119800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198015020 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1198011d10 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a4199330 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119800f2e0 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.637+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a41a54f0 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.638+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a406a420 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.639+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1198018020 con 0x7f11a4101800 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.640+0000 7f118bfff700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f118c0a2cb0 0x7f118c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.640+0000 7f11a2ffd700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f118c0a2cb0 0x7f118c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:06.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.640+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f11980cf090 con 0x7f11a4101800 2024-04-22T01:12:06.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.640+0000 7f11a2ffd700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f118c0a2cb0 0x7f118c0a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1194005950 tx=0x7f119400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:06.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.644+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f119801f020 con 0x7f11a4101800 2024-04-22T01:12:06.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.856+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f11a402db30 con 0x7f11a4101800 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3050425563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:12:06.857 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.856+0000 7f118bfff700 1 -- 172.21.15.38:0/1566281993 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f119809bc40 con 0x7f11a4101800 2024-04-22T01:12:06.857 INFO:teuthology.orchestra.run.smithi038.stdout:55834574933 2024-04-22T01:12:06.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f118c0a2cb0 msgr2=0x7f118c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f118c0a2cb0 0x7f118c0a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1194005950 tx=0x7f119400b410 comp rx=0 tx=0).stop 2024-04-22T01:12:06.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 msgr2=0x7f11a419ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 0x7f11a419ebd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f119800b530 tx=0x7f119800b840 comp rx=0 tx=0).stop 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 shutdown_connections 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f118c0a2cb0 0x7f118c0a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11a4107670 0x7f11a4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a4100e50 0x7f11a419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 --2- 172.21.15.38:0/1566281993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11a4101800 0x7f11a419ebd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 >> 172.21.15.38:0/1566281993 conn(0x7f11a4076450 msgr2=0x7f11a40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 shutdown_connections 2024-04-22T01:12:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:06.859+0000 7f11a95be700 1 -- 172.21.15.38:0/1566281993 wait complete. 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3050425563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:12:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:07.132 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182469 got 107374182474 for osd.3 2024-04-22T01:12:07.132 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3050425563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:12:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:12:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:07 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1566281993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T01:12:08.128 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:07 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1566281993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T01:12:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:07 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1566281993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T01:12:08.895 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574933 for osd.1 2024-04-22T01:12:08.895 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:08 smithi203 ceph-mon[31825]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:09.026 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:08 smithi038 ceph-mon[28043]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:09.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 -- 172.21.15.38:0/2353929134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0564100fc0 msgr2=0x7f0564108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:09.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 --2- 172.21.15.38:0/2353929134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0564100fc0 0x7f0564108010 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f05580099c0 tx=0x7f0558009cd0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 -- 172.21.15.38:0/2353929134 shutdown_connections 2024-04-22T01:12:09.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 --2- 172.21.15.38:0/2353929134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05641086f0 0x7f056410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 --2- 172.21.15.38:0/2353929134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0564100610 0x7f05641009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 --2- 172.21.15.38:0/2353929134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0564100fc0 0x7f0564108010 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.142+0000 7f05691a8700 1 -- 172.21.15.38:0/2353929134 >> 172.21.15.38:0/2353929134 conn(0x7f0564077330 msgr2=0x7f05640fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:09.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.143+0000 7f05691a8700 1 -- 172.21.15.38:0/2353929134 shutdown_connections 2024-04-22T01:12:09.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.143+0000 7f05691a8700 1 -- 172.21.15.38:0/2353929134 wait complete. 2024-04-22T01:12:09.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.143+0000 7f05691a8700 1 Processor -- start 2024-04-22T01:12:09.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.143+0000 7f05691a8700 1 -- start start 2024-04-22T01:12:09.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f05691a8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0564100610 0x7f056419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:09.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f05691a8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0564100fc0 0x7f056419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:09.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f05691a8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 0x7f0564198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f05691a8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0564112590 con 0x7f05641086f0 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f05691a8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0564112410 con 0x7f0564100610 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f05691a8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0564112710 con 0x7f0564100fc0 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 0x7f0564198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 0x7f0564198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52280/0 (socket says 172.21.15.38:52280) 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 -- 172.21.15.38:0/1985907599 learned_addr learned my addr 172.21.15.38:0/1985907599 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f0562d9d700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0564100610 0x7f056419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056259c700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0564100fc0 0x7f056419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 -- 172.21.15.38:0/1985907599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0564100fc0 msgr2=0x7f056419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0564100fc0 0x7f056419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 -- 172.21.15.38:0/1985907599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0564100610 msgr2=0x7f056419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0564100610 0x7f056419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 -- 172.21.15.38:0/1985907599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0564199010 con 0x7f05641086f0 2024-04-22T01:12:09.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.144+0000 7f056359e700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 0x7f0564198830 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f055c007d70 tx=0x7f055c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:09.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.145+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c016020 con 0x7f05641086f0 2024-04-22T01:12:09.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.145+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c00ac90 con 0x7f05641086f0 2024-04-22T01:12:09.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.145+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0564199300 con 0x7f05641086f0 2024-04-22T01:12:09.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.145+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0115c0 con 0x7f05641086f0 2024-04-22T01:12:09.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.145+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05641a5510 con 0x7f05641086f0 2024-04-22T01:12:09.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:08 smithi111 ceph-mon[31057]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:09.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.146+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0564101740 con 0x7f05641086f0 2024-04-22T01:12:09.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.148+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f055c019020 con 0x7f05641086f0 2024-04-22T01:12:09.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.148+0000 7f054bfff700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f054c0a2cb0 0x7f054c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:09.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.148+0000 7f0562d9d700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f054c0a2cb0 0x7f054c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:09.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.148+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f055c0cf790 con 0x7f05641086f0 2024-04-22T01:12:09.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.149+0000 7f0562d9d700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f054c0a2cb0 0x7f054c0a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0554009a10 tx=0x7f0554009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:09.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.152+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f055c020020 con 0x7f05641086f0 2024-04-22T01:12:09.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.358+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f056402db30 con 0x7f05641086f0 2024-04-22T01:12:09.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.359+0000 7f054bfff700 1 -- 172.21.15.38:0/1985907599 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f055c09c340 con 0x7f05641086f0 2024-04-22T01:12:09.358 INFO:teuthology.orchestra.run.smithi038.stdout:287762808876 2024-04-22T01:12:09.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f054c0a2cb0 msgr2=0x7f054c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:09.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f054c0a2cb0 0x7f054c0a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0554009a10 tx=0x7f0554009190 comp rx=0 tx=0).stop 2024-04-22T01:12:09.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 msgr2=0x7f0564198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:09.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 0x7f0564198830 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f055c007d70 tx=0x7f055c00a340 comp rx=0 tx=0).stop 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 shutdown_connections 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f054c0a2cb0 0x7f054c0a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05641086f0 0x7f0564198830 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0564100610 0x7f056419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 --2- 172.21.15.38:0/1985907599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0564100fc0 0x7f056419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 >> 172.21.15.38:0/1985907599 conn(0x7f0564077330 msgr2=0x7f05640fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 shutdown_connections 2024-04-22T01:12:09.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:09.361+0000 7f05691a8700 1 -- 172.21.15.38:0/1985907599 wait complete. 2024-04-22T01:12:10.146 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:09 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1985907599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T01:12:10.160 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:09 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1985907599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T01:12:10.220 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808876 for osd.9 2024-04-22T01:12:10.220 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:10.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:09 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1985907599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 -- 172.21.15.38:0/392673021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 msgr2=0x7f732c101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 --2- 172.21.15.38:0/392673021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c101ab0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7328009990 tx=0x7f7328009ca0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 -- 172.21.15.38:0/392673021 shutdown_connections 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 --2- 172.21.15.38:0/392673021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f732c106770 0x7f732c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 --2- 172.21.15.38:0/392673021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c101ab0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 --2- 172.21.15.38:0/392673021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f732c105dd0 0x7f732c106230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 -- 172.21.15.38:0/392673021 >> 172.21.15.38:0/392673021 conn(0x7f732c077ff0 msgr2=0x7f732c078400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 -- 172.21.15.38:0/392673021 shutdown_connections 2024-04-22T01:12:10.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.455+0000 7f733395f700 1 -- 172.21.15.38:0/392673021 wait complete. 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 Processor -- start 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 -- start start 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f732c105dd0 0x7f732c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f732c106770 0x7f732c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f732c111450 con 0x7f732c1016d0 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f732c1112d0 con 0x7f732c106770 2024-04-22T01:12:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f733395f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f732c1115d0 con 0x7f732c105dd0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f73316fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f73316fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52296/0 (socket says 172.21.15.38:52296) 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f73316fb700 1 -- 172.21.15.38:0/4281972509 learned_addr learned my addr 172.21.15.38:0/4281972509 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f7331efc700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f732c106770 0x7f732c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.456+0000 7f7330efa700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f732c105dd0 0x7f732c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73316fb700 1 -- 172.21.15.38:0/4281972509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f732c105dd0 msgr2=0x7f732c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73316fb700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f732c105dd0 0x7f732c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73316fb700 1 -- 172.21.15.38:0/4281972509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f732c106770 msgr2=0x7f732c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73316fb700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f732c106770 0x7f732c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73316fb700 1 -- 172.21.15.38:0/4281972509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c198ff0 con 0x7f732c1016d0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73316fb700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c19e690 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7328000c00 tx=0x7f73280047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328027070 con 0x7f732c1016d0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732c199280 con 0x7f732c1016d0 2024-04-22T01:12:10.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7328003e40 con 0x7f732c1016d0 2024-04-22T01:12:10.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328022d10 con 0x7f732c1016d0 2024-04-22T01:12:10.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.457+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732c1a54f0 con 0x7f732c1016d0 2024-04-22T01:12:10.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.459+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7310005330 con 0x7f732c1016d0 2024-04-22T01:12:10.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.460+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f732802a070 con 0x7f732c1016d0 2024-04-22T01:12:10.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.460+0000 7f73227fc700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73180a2d30 0x7f73180a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:10.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.460+0000 7f7330efa700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73180a2d30 0x7f73180a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:10.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.460+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f73280d01a0 con 0x7f732c1016d0 2024-04-22T01:12:10.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.461+0000 7f7330efa700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73180a2d30 0x7f73180a51f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f732c078740 tx=0x7f731c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:10.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.465+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f732805e6f0 con 0x7f732c1016d0 2024-04-22T01:12:10.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.677+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f73100058f0 con 0x7f732c1016d0 2024-04-22T01:12:10.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.678+0000 7f73227fc700 1 -- 172.21.15.38:0/4281972509 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7328013070 con 0x7f732c1016d0 2024-04-22T01:12:10.677 INFO:teuthology.orchestra.run.smithi038.stdout:309237645349 2024-04-22T01:12:10.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73180a2d30 msgr2=0x7f73180a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73180a2d30 0x7f73180a51f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f732c078740 tx=0x7f731c00b3c0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 msgr2=0x7f732c19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c19e690 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7328000c00 tx=0x7f73280047b0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 shutdown_connections 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73180a2d30 0x7f73180a51f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f732c106770 0x7f732c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f732c1016d0 0x7f732c19e690 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 --2- 172.21.15.38:0/4281972509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f732c105dd0 0x7f732c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 >> 172.21.15.38:0/4281972509 conn(0x7f732c077ff0 msgr2=0x7f732c06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 shutdown_connections 2024-04-22T01:12:10.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:10.680+0000 7f733395f700 1 -- 172.21.15.38:0/4281972509 wait complete. 2024-04-22T01:12:11.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:10 smithi038 ceph-mon[28043]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:11.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:10 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4281972509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T01:12:11.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:10 smithi111 ceph-mon[31057]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:11.064 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:10 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4281972509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T01:12:11.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:10 smithi203 ceph-mon[31825]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:11.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:10 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4281972509' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T01:12:11.480 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645349 for osd.10 2024-04-22T01:12:11.480 DEBUG:teuthology.parallel:result is None 2024-04-22T01:12:11.480 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T01:12:11.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T01:12:11.737 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:13.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:12 smithi111 ceph-mon[31057]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:12:13.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:12 smithi038 ceph-mon[28043]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:12:13.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:12 smithi203 ceph-mon[31825]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 -- 172.21.15.38:0/1584245961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406b360 msgr2=0x7f087410f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 --2- 172.21.15.38:0/1584245961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406b360 0x7f087410f100 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f086c00b0d0 tx=0x7f086c00b3e0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 -- 172.21.15.38:0/1584245961 shutdown_connections 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 --2- 172.21.15.38:0/1584245961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406b360 0x7f087410f100 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 --2- 172.21.15.38:0/1584245961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08741082c0 0x7f08741086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 --2- 172.21.15.38:0/1584245961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f087406a9c0 0x7f087406ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 -- 172.21.15.38:0/1584245961 >> 172.21.15.38:0/1584245961 conn(0x7f0874078060 msgr2=0x7f0874078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 -- 172.21.15.38:0/1584245961 shutdown_connections 2024-04-22T01:12:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.608+0000 7f087c64f700 1 -- 172.21.15.38:0/1584245961 wait complete. 2024-04-22T01:12:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 Processor -- start 2024-04-22T01:12:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 -- start start 2024-04-22T01:12:13.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 0x7f0874101240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:13.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f087406b360 0x7f0874101780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:13.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08741082c0 0x7f0874101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:13.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0874112510 con 0x7f087406a9c0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0874112390 con 0x7f08741082c0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087c64f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0874112690 con 0x7f087406b360 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087a3eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 0x7f0874101240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087a3eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 0x7f0874101240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52308/0 (socket says 172.21.15.38:52308) 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f087a3eb700 1 -- 172.21.15.38:0/3287301565 learned_addr learned my addr 172.21.15.38:0/3287301565 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.609+0000 7f0879bea700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f087406b360 0x7f0874101780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087abec700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08741082c0 0x7f0874101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087a3eb700 1 -- 172.21.15.38:0/3287301565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f087406b360 msgr2=0x7f0874101780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087a3eb700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f087406b360 0x7f0874101780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087a3eb700 1 -- 172.21.15.38:0/3287301565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08741082c0 msgr2=0x7f0874101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087a3eb700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08741082c0 0x7f0874101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087a3eb700 1 -- 172.21.15.38:0/3287301565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08741a6f10 con 0x7f087406a9c0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087abec700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08741082c0 0x7f0874101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087a3eb700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 0x7f0874101240 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f087000c9b0 tx=0x7f087000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870007b00 con 0x7f087406a9c0 2024-04-22T01:12:13.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0870007ca0 con 0x7f087406a9c0 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08741a7200 con 0x7f087406a9c0 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870018730 con 0x7f087406a9c0 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.610+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08741a7740 con 0x7f087406a9c0 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.611+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0874103970 con 0x7f087406a9c0 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.612+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f08700188d0 con 0x7f087406a9c0 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.613+0000 7f08677fe700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f08600a2cb0 0x7f08600a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:13.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.613+0000 7f0879bea700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f08600a2cb0 0x7f08600a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:13.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.613+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f08700cf4c0 con 0x7f087406a9c0 2024-04-22T01:12:13.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.613+0000 7f0879bea700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f08600a2cb0 0x7f08600a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08740786a0 tx=0x7f086800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:13.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.617+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0870014030 con 0x7f087406a9c0 2024-04-22T01:12:13.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.817+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0874102770 con 0x7f08600a2cb0 2024-04-22T01:12:13.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.827+0000 7f08677fe700 1 -- 172.21.15.38:0/3287301565 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340675 (secure 0 0 0) 0x7f0874102770 con 0x7f08600a2cb0 2024-04-22T01:12:13.832 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:13.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f08600a2cb0 msgr2=0x7f08600a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:13.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f08600a2cb0 0x7f08600a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08740786a0 tx=0x7f086800b410 comp rx=0 tx=0).stop 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 msgr2=0x7f0874101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 0x7f0874101240 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f087000c9b0 tx=0x7f087000ccc0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 shutdown_connections 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f08600a2cb0 0x7f08600a5170 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08741082c0 0x7f0874101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f087406a9c0 0x7f0874101240 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 --2- 172.21.15.38:0/3287301565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f087406b360 0x7f0874101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 >> 172.21.15.38:0/3287301565 conn(0x7f0874078060 msgr2=0x7f08740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 shutdown_connections 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:13.832+0000 7f087c64f700 1 -- 172.21.15.38:0/3287301565 wait complete. 2024-04-22T01:12:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:12:14.646 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-04-22T01:12:13.811464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658448,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":876,"num_read_kb":561,"num_write":421,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":347,"ondisk_log_size":347,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357372,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584452,"statfs":{"total":1151940427776,"available":1151574478848,"internally_reserved":0,"allocated":12591104,"data_stored":7961246,"data_compressed":356811,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003297"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837343+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:11:17.837343+0000","last_peered":"2024-04-22T01:11:17.837343+0000","last_clean":"2024-04-22T01:11:17.837343+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:11:17.837343+0000","last_undegraded":"2024-04-22T01:11:17.837343+0000","last_fullsized":"2024-04-22T01:11:17.837343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308435+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:11:17.308435+0000","last_peered":"2024-04-22T01:11:17.308435+0000","last_clean":"2024-04-22T01:11:17.308435+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:11:17.308435+0000","last_undegraded":"2024-04-22T01:11:17.308435+0000","last_fullsized":"2024-04-22T01:11:17.308435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308058+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:11:17.308058+0000","last_peered":"2024-04-22T01:11:17.308058+0000","last_clean":"2024-04-22T01:11:17.308058+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:11:17.308058+0000","last_undegraded":"2024-04-22T01:11:17.308058+0000","last_fullsized":"2024-04-22T01:11:17.308058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306007+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:11:17.306007+0000","last_peered":"2024-04-22T01:11:17.306007+0000","last_clean":"2024-04-22T01:11:17.306007+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:11:17.306007+0000","last_undegraded":"2024-04-22T01:11:17.306007+0000","last_fullsized":"2024-04-22T01:11:17.306007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584434+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:11:17.584434+0000","last_peered":"2024-04-22T01:11:17.584434+0000","last_clean":"2024-04-22T01:11:17.584434+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:11:17.584434+0000","last_undegraded":"2024-04-22T01:11:17.584434+0000","last_fullsized":"2024-04-22T01:11:17.584434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307537+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:11:17.307537+0000","last_peered":"2024-04-22T01:11:17.307537+0000","last_clean":"2024-04-22T01:11:17.307537+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:11:17.307537+0000","last_undegraded":"2024-04-22T01:11:17.307537+0000","last_fullsized":"2024-04-22T01:11:17.307537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311342+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:11:17.311342+0000","last_peered":"2024-04-22T01:11:17.311342+0000","last_clean":"2024-04-22T01:11:17.311342+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:11:17.311342+0000","last_undegraded":"2024-04-22T01:11:17.311342+0000","last_fullsized":"2024-04-22T01:11:17.311342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.522657+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:11:17.522657+0000","last_peered":"2024-04-22T01:11:17.522657+0000","last_clean":"2024-04-22T01:11:17.522657+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:11:17.522657+0000","last_undegraded":"2024-04-22T01:11:17.522657+0000","last_fullsized":"2024-04-22T01:11:17.522657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310180+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:11:17.310180+0000","last_peered":"2024-04-22T01:11:17.310180+0000","last_clean":"2024-04-22T01:11:17.310180+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:11:17.310180+0000","last_undegraded":"2024-04-22T01:11:17.310180+0000","last_fullsized":"2024-04-22T01:11:17.310180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308084+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:11:17.308084+0000","last_peered":"2024-04-22T01:11:17.308084+0000","last_clean":"2024-04-22T01:11:17.308084+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:11:17.308084+0000","last_undegraded":"2024-04-22T01:11:17.308084+0000","last_fullsized":"2024-04-22T01:11:17.308084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312398+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:11:17.312398+0000","last_peered":"2024-04-22T01:11:17.312398+0000","last_clean":"2024-04-22T01:11:17.312398+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:11:17.312398+0000","last_undegraded":"2024-04-22T01:11:17.312398+0000","last_fullsized":"2024-04-22T01:11:17.312398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305890+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:11:17.305890+0000","last_peered":"2024-04-22T01:11:17.305890+0000","last_clean":"2024-04-22T01:11:17.305890+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:11:17.305890+0000","last_undegraded":"2024-04-22T01:11:17.305890+0000","last_fullsized":"2024-04-22T01:11:17.305890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312524+0000","last_change":"2024-04-22T01:11:17.312524+0000","last_active":"2024-04-22T01:11:17.312524+0000","last_peered":"2024-04-22T01:11:17.312524+0000","last_clean":"2024-04-22T01:11:17.312524+0000","last_became_active":"2024-04-22T01:11:17.312068+0000","last_became_peered":"2024-04-22T01:11:17.312068+0000","last_unstale":"2024-04-22T01:11:17.312524+0000","last_undegraded":"2024-04-22T01:11:17.312524+0000","last_fullsized":"2024-04-22T01:11:17.312524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:47:17.243712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307962+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:11:17.307962+0000","last_peered":"2024-04-22T01:11:17.307962+0000","last_clean":"2024-04-22T01:11:17.307962+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:11:17.307962+0000","last_undegraded":"2024-04-22T01:11:17.307962+0000","last_fullsized":"2024-04-22T01:11:17.307962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308200+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:11:17.308200+0000","last_peered":"2024-04-22T01:11:17.308200+0000","last_clean":"2024-04-22T01:11:17.308200+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:11:17.308200+0000","last_undegraded":"2024-04-22T01:11:17.308200+0000","last_fullsized":"2024-04-22T01:11:17.308200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.309193+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:11:17.309193+0000","last_peered":"2024-04-22T01:11:17.309193+0000","last_clean":"2024-04-22T01:11:17.309193+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:11:17.309193+0000","last_undegraded":"2024-04-22T01:11:17.309193+0000","last_fullsized":"2024-04-22T01:11:17.309193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837232+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:11:17.837232+0000","last_peered":"2024-04-22T01:11:17.837232+0000","last_clean":"2024-04-22T01:11:17.837232+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:11:17.837232+0000","last_undegraded":"2024-04-22T01:11:17.837232+0000","last_fullsized":"2024-04-22T01:11:17.837232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307085+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:11:17.307085+0000","last_peered":"2024-04-22T01:11:17.307085+0000","last_clean":"2024-04-22T01:11:17.307085+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:11:17.307085+0000","last_undegraded":"2024-04-22T01:11:17.307085+0000","last_fullsized":"2024-04-22T01:11:17.307085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305898+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:11:17.305898+0000","last_peered":"2024-04-22T01:11:17.305898+0000","last_clean":"2024-04-22T01:11:17.305898+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:11:17.305898+0000","last_undegraded":"2024-04-22T01:11:17.305898+0000","last_fullsized":"2024-04-22T01:11:17.305898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586121+0000","last_change":"2024-04-22T01:11:17.586121+0000","last_active":"2024-04-22T01:11:17.586121+0000","last_peered":"2024-04-22T01:11:17.586121+0000","last_clean":"2024-04-22T01:11:17.586121+0000","last_became_active":"2024-04-22T01:11:17.585619+0000","last_became_peered":"2024-04-22T01:11:17.585619+0000","last_unstale":"2024-04-22T01:11:17.586121+0000","last_undegraded":"2024-04-22T01:11:17.586121+0000","last_fullsized":"2024-04-22T01:11:17.586121+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:12.987810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:09.536471+0000","last_change":"2024-04-22T01:10:29.585069+0000","last_active":"2024-04-22T01:12:09.536471+0000","last_peered":"2024-04-22T01:12:09.536471+0000","last_clean":"2024-04-22T01:12:09.536471+0000","last_became_active":"2024-04-22T01:10:29.584651+0000","last_became_peered":"2024-04-22T01:10:29.584651+0000","last_unstale":"2024-04-22T01:12:09.536471+0000","last_undegraded":"2024-04-22T01:12:09.536471+0000","last_fullsized":"2024-04-22T01:12:09.536471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:13.121214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523254+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:11:17.523254+0000","last_peered":"2024-04-22T01:11:17.523254+0000","last_clean":"2024-04-22T01:11:17.523254+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:11:17.523254+0000","last_undegraded":"2024-04-22T01:11:17.523254+0000","last_fullsized":"2024-04-22T01:11:17.523254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305110+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:11:17.305110+0000","last_peered":"2024-04-22T01:11:17.305110+0000","last_clean":"2024-04-22T01:11:17.305110+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:11:17.305110+0000","last_undegraded":"2024-04-22T01:11:17.305110+0000","last_fullsized":"2024-04-22T01:11:17.305110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523715+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:11:17.523715+0000","last_peered":"2024-04-22T01:11:17.523715+0000","last_clean":"2024-04-22T01:11:17.523715+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:11:17.523715+0000","last_undegraded":"2024-04-22T01:11:17.523715+0000","last_fullsized":"2024-04-22T01:11:17.523715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308307+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:11:17.308307+0000","last_peered":"2024-04-22T01:11:17.308307+0000","last_clean":"2024-04-22T01:11:17.308307+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:11:17.308307+0000","last_undegraded":"2024-04-22T01:11:17.308307+0000","last_fullsized":"2024-04-22T01:11:17.308307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305419+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:11:17.305419+0000","last_peered":"2024-04-22T01:11:17.305419+0000","last_clean":"2024-04-22T01:11:17.305419+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:11:17.305419+0000","last_undegraded":"2024-04-22T01:11:17.305419+0000","last_fullsized":"2024-04-22T01:11:17.305419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584910+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:11:17.584910+0000","last_peered":"2024-04-22T01:11:17.584910+0000","last_clean":"2024-04-22T01:11:17.584910+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:11:17.584910+0000","last_undegraded":"2024-04-22T01:11:17.584910+0000","last_fullsized":"2024-04-22T01:11:17.584910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.309976+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:11:17.309976+0000","last_peered":"2024-04-22T01:11:17.309976+0000","last_clean":"2024-04-22T01:11:17.309976+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:11:17.309976+0000","last_undegraded":"2024-04-22T01:11:17.309976+0000","last_fullsized":"2024-04-22T01:11:17.309976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305145+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:11:17.305145+0000","last_peered":"2024-04-22T01:11:17.305145+0000","last_clean":"2024-04-22T01:11:17.305145+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:11:17.305145+0000","last_undegraded":"2024-04-22T01:11:17.305145+0000","last_fullsized":"2024-04-22T01:11:17.305145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584383+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:11:17.584383+0000","last_peered":"2024-04-22T01:11:17.584383+0000","last_clean":"2024-04-22T01:11:17.584383+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:11:17.584383+0000","last_undegraded":"2024-04-22T01:11:17.584383+0000","last_fullsized":"2024-04-22T01:11:17.584383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307891+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:11:17.307891+0000","last_peered":"2024-04-22T01:11:17.307891+0000","last_clean":"2024-04-22T01:11:17.307891+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:11:17.307891+0000","last_undegraded":"2024-04-22T01:11:17.307891+0000","last_fullsized":"2024-04-22T01:11:17.307891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305124+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:11:17.305124+0000","last_peered":"2024-04-22T01:11:17.305124+0000","last_clean":"2024-04-22T01:11:17.305124+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:11:17.305124+0000","last_undegraded":"2024-04-22T01:11:17.305124+0000","last_fullsized":"2024-04-22T01:11:17.305124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:08.536404+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:12:08.536404+0000","last_peered":"2024-04-22T01:12:08.536404+0000","last_clean":"2024-04-22T01:12:08.536404+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:12:08.536404+0000","last_undegraded":"2024-04-22T01:12:08.536404+0000","last_fullsized":"2024-04-22T01:12:08.536404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584145+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:11:17.584145+0000","last_peered":"2024-04-22T01:11:17.584145+0000","last_clean":"2024-04-22T01:11:17.584145+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:11:17.584145+0000","last_undegraded":"2024-04-22T01:11:17.584145+0000","last_fullsized":"2024-04-22T01:11:17.584145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305097+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:11:17.305097+0000","last_peered":"2024-04-22T01:11:17.305097+0000","last_clean":"2024-04-22T01:11:17.305097+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:11:17.305097+0000","last_undegraded":"2024-04-22T01:11:17.305097+0000","last_fullsized":"2024-04-22T01:11:17.305097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307542+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:11:17.307542+0000","last_peered":"2024-04-22T01:11:17.307542+0000","last_clean":"2024-04-22T01:11:17.307542+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:11:17.307542+0000","last_undegraded":"2024-04-22T01:11:17.307542+0000","last_fullsized":"2024-04-22T01:11:17.307542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307530+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:11:17.307530+0000","last_peered":"2024-04-22T01:11:17.307530+0000","last_clean":"2024-04-22T01:11:17.307530+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:11:17.307530+0000","last_undegraded":"2024-04-22T01:11:17.307530+0000","last_fullsized":"2024-04-22T01:11:17.307530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837202+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:11:17.837202+0000","last_peered":"2024-04-22T01:11:17.837202+0000","last_clean":"2024-04-22T01:11:17.837202+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:11:17.837202+0000","last_undegraded":"2024-04-22T01:11:17.837202+0000","last_fullsized":"2024-04-22T01:11:17.837202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310483+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:11:17.310483+0000","last_peered":"2024-04-22T01:11:17.310483+0000","last_clean":"2024-04-22T01:11:17.310483+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:11:17.310483+0000","last_undegraded":"2024-04-22T01:11:17.310483+0000","last_fullsized":"2024-04-22T01:11:17.310483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.522957+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:11:17.522957+0000","last_peered":"2024-04-22T01:11:17.522957+0000","last_clean":"2024-04-22T01:11:17.522957+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:11:17.522957+0000","last_undegraded":"2024-04-22T01:11:17.522957+0000","last_fullsized":"2024-04-22T01:11:17.522957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308289+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:11:17.308289+0000","last_peered":"2024-04-22T01:11:17.308289+0000","last_clean":"2024-04-22T01:11:17.308289+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:11:17.308289+0000","last_undegraded":"2024-04-22T01:11:17.308289+0000","last_fullsized":"2024-04-22T01:11:17.308289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307754+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:11:17.307754+0000","last_peered":"2024-04-22T01:11:17.307754+0000","last_clean":"2024-04-22T01:11:17.307754+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:11:17.307754+0000","last_undegraded":"2024-04-22T01:11:17.307754+0000","last_fullsized":"2024-04-22T01:11:17.307754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306199+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:11:17.306199+0000","last_peered":"2024-04-22T01:11:17.306199+0000","last_clean":"2024-04-22T01:11:17.306199+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:11:17.306199+0000","last_undegraded":"2024-04-22T01:11:17.306199+0000","last_fullsized":"2024-04-22T01:11:17.306199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523060+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:11:17.523060+0000","last_peered":"2024-04-22T01:11:17.523060+0000","last_clean":"2024-04-22T01:11:17.523060+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:11:17.523060+0000","last_undegraded":"2024-04-22T01:11:17.523060+0000","last_fullsized":"2024-04-22T01:11:17.523060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:11.539982+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:12:11.539982+0000","last_peered":"2024-04-22T01:12:11.539982+0000","last_clean":"2024-04-22T01:12:11.539982+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:12:11.539982+0000","last_undegraded":"2024-04-22T01:12:11.539982+0000","last_fullsized":"2024-04-22T01:12:11.539982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310305+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:11:17.310305+0000","last_peered":"2024-04-22T01:11:17.310305+0000","last_clean":"2024-04-22T01:11:17.310305+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:11:17.310305+0000","last_undegraded":"2024-04-22T01:11:17.310305+0000","last_fullsized":"2024-04-22T01:11:17.310305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306083+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:11:17.306083+0000","last_peered":"2024-04-22T01:11:17.306083+0000","last_clean":"2024-04-22T01:11:17.306083+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:11:17.306083+0000","last_undegraded":"2024-04-22T01:11:17.306083+0000","last_fullsized":"2024-04-22T01:11:17.306083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312678+0000","last_change":"2024-04-22T01:10:29.591793+0000","last_active":"2024-04-22T01:11:17.312678+0000","last_peered":"2024-04-22T01:11:17.312678+0000","last_clean":"2024-04-22T01:11:17.312678+0000","last_became_active":"2024-04-22T01:10:29.591245+0000","last_became_peered":"2024-04-22T01:10:29.591245+0000","last_unstale":"2024-04-22T01:11:17.312678+0000","last_undegraded":"2024-04-22T01:11:17.312678+0000","last_fullsized":"2024-04-22T01:11:17.312678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:40:28.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308167+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:11:17.308167+0000","last_peered":"2024-04-22T01:11:17.308167+0000","last_clean":"2024-04-22T01:11:17.308167+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:11:17.308167+0000","last_undegraded":"2024-04-22T01:11:17.308167+0000","last_fullsized":"2024-04-22T01:11:17.308167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586221+0000","last_change":"2024-04-22T01:11:17.586221+0000","last_active":"2024-04-22T01:11:17.586221+0000","last_peered":"2024-04-22T01:11:17.586221+0000","last_clean":"2024-04-22T01:11:17.586221+0000","last_became_active":"2024-04-22T01:11:17.585689+0000","last_became_peered":"2024-04-22T01:11:17.585689+0000","last_unstale":"2024-04-22T01:11:17.586221+0000","last_undegraded":"2024-04-22T01:11:17.586221+0000","last_fullsized":"2024-04-22T01:11:17.586221+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:40.327272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307759+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:11:17.307759+0000","last_peered":"2024-04-22T01:11:17.307759+0000","last_clean":"2024-04-22T01:11:17.307759+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:11:17.307759+0000","last_undegraded":"2024-04-22T01:11:17.307759+0000","last_fullsized":"2024-04-22T01:11:17.307759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.585668+0000","last_change":"2024-04-22T01:11:17.585668+0000","last_active":"2024-04-22T01:11:17.585668+0000","last_peered":"2024-04-22T01:11:17.585668+0000","last_clean":"2024-04-22T01:11:17.585668+0000","last_became_active":"2024-04-22T01:11:17.585217+0000","last_became_peered":"2024-04-22T01:11:17.585217+0000","last_unstale":"2024-04-22T01:11:17.585668+0000","last_undegraded":"2024-04-22T01:11:17.585668+0000","last_fullsized":"2024-04-22T01:11:17.585668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:24.007210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306370+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:11:17.306370+0000","last_peered":"2024-04-22T01:11:17.306370+0000","last_clean":"2024-04-22T01:11:17.306370+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:11:17.306370+0000","last_undegraded":"2024-04-22T01:11:17.306370+0000","last_fullsized":"2024-04-22T01:11:17.306370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":209,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:11.546386+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:12:11.546386+0000","last_peered":"2024-04-22T01:12:11.546386+0000","last_clean":"2024-04-22T01:12:11.546386+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:12:11.546386+0000","last_undegraded":"2024-04-22T01:12:11.546386+0000","last_fullsized":"2024-04-22T01:12:11.546386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60642,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":150,"num_read_kb":134,"num_write":35,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584635+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:11:17.584635+0000","last_peered":"2024-04-22T01:11:17.584635+0000","last_clean":"2024-04-22T01:11:17.584635+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:11:17.584635+0000","last_undegraded":"2024-04-22T01:11:17.584635+0000","last_fullsized":"2024-04-22T01:11:17.584635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308625+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:11:17.308625+0000","last_peered":"2024-04-22T01:11:17.308625+0000","last_clean":"2024-04-22T01:11:17.308625+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:11:17.308625+0000","last_undegraded":"2024-04-22T01:11:17.308625+0000","last_fullsized":"2024-04-22T01:11:17.308625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305262+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:11:17.305262+0000","last_peered":"2024-04-22T01:11:17.305262+0000","last_clean":"2024-04-22T01:11:17.305262+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:11:17.305262+0000","last_undegraded":"2024-04-22T01:11:17.305262+0000","last_fullsized":"2024-04-22T01:11:17.305262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310549+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:11:17.310549+0000","last_peered":"2024-04-22T01:11:17.310549+0000","last_clean":"2024-04-22T01:11:17.310549+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:11:17.310549+0000","last_undegraded":"2024-04-22T01:11:17.310549+0000","last_fullsized":"2024-04-22T01:11:17.310549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836995+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:11:17.836995+0000","last_peered":"2024-04-22T01:11:17.836995+0000","last_clean":"2024-04-22T01:11:17.836995+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:11:17.836995+0000","last_undegraded":"2024-04-22T01:11:17.836995+0000","last_fullsized":"2024-04-22T01:11:17.836995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307337+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:11:17.307337+0000","last_peered":"2024-04-22T01:11:17.307337+0000","last_clean":"2024-04-22T01:11:17.307337+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:11:17.307337+0000","last_undegraded":"2024-04-22T01:11:17.307337+0000","last_fullsized":"2024-04-22T01:11:17.307337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311538+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:11:17.311538+0000","last_peered":"2024-04-22T01:11:17.311538+0000","last_clean":"2024-04-22T01:11:17.311538+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:11:17.311538+0000","last_undegraded":"2024-04-22T01:11:17.311538+0000","last_fullsized":"2024-04-22T01:11:17.311538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306262+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:11:17.306262+0000","last_peered":"2024-04-22T01:11:17.306262+0000","last_clean":"2024-04-22T01:11:17.306262+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:11:17.306262+0000","last_undegraded":"2024-04-22T01:11:17.306262+0000","last_fullsized":"2024-04-22T01:11:17.306262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:09.562614+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:12:09.562614+0000","last_peered":"2024-04-22T01:12:09.562614+0000","last_clean":"2024-04-22T01:12:09.562614+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:12:09.562614+0000","last_undegraded":"2024-04-22T01:12:09.562614+0000","last_fullsized":"2024-04-22T01:12:09.562614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.314098+0000","last_change":"2024-04-22T01:11:17.314098+0000","last_active":"2024-04-22T01:11:17.314098+0000","last_peered":"2024-04-22T01:11:17.314098+0000","last_clean":"2024-04-22T01:11:17.314098+0000","last_became_active":"2024-04-22T01:11:17.313663+0000","last_became_peered":"2024-04-22T01:11:17.313663+0000","last_unstale":"2024-04-22T01:11:17.314098+0000","last_undegraded":"2024-04-22T01:11:17.314098+0000","last_fullsized":"2024-04-22T01:11:17.314098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:20:03.826857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310992+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:11:17.310992+0000","last_peered":"2024-04-22T01:11:17.310992+0000","last_clean":"2024-04-22T01:11:17.310992+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:11:17.310992+0000","last_undegraded":"2024-04-22T01:11:17.310992+0000","last_fullsized":"2024-04-22T01:11:17.310992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584269+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:11:17.584269+0000","last_peered":"2024-04-22T01:11:17.584269+0000","last_clean":"2024-04-22T01:11:17.584269+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:11:17.584269+0000","last_undegraded":"2024-04-22T01:11:17.584269+0000","last_fullsized":"2024-04-22T01:11:17.584269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307005+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:11:17.307005+0000","last_peered":"2024-04-22T01:11:17.307005+0000","last_clean":"2024-04-22T01:11:17.307005+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:11:17.307005+0000","last_undegraded":"2024-04-22T01:11:17.307005+0000","last_fullsized":"2024-04-22T01:11:17.307005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307000+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:11:17.307000+0000","last_peered":"2024-04-22T01:11:17.307000+0000","last_clean":"2024-04-22T01:11:17.307000+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:11:17.307000+0000","last_undegraded":"2024-04-22T01:11:17.307000+0000","last_fullsized":"2024-04-22T01:11:17.307000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584776+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:11:17.584776+0000","last_peered":"2024-04-22T01:11:17.584776+0000","last_clean":"2024-04-22T01:11:17.584776+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:11:17.584776+0000","last_undegraded":"2024-04-22T01:11:17.584776+0000","last_fullsized":"2024-04-22T01:11:17.584776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837145+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:11:17.837145+0000","last_peered":"2024-04-22T01:11:17.837145+0000","last_clean":"2024-04-22T01:11:17.837145+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:11:17.837145+0000","last_undegraded":"2024-04-22T01:11:17.837145+0000","last_fullsized":"2024-04-22T01:11:17.837145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584360+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:11:17.584360+0000","last_peered":"2024-04-22T01:11:17.584360+0000","last_clean":"2024-04-22T01:11:17.584360+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:11:17.584360+0000","last_undegraded":"2024-04-22T01:11:17.584360+0000","last_fullsized":"2024-04-22T01:11:17.584360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306511+0000","last_change":"2024-04-22T01:10:29.590388+0000","last_active":"2024-04-22T01:11:17.306511+0000","last_peered":"2024-04-22T01:11:17.306511+0000","last_clean":"2024-04-22T01:11:17.306511+0000","last_became_active":"2024-04-22T01:10:29.590035+0000","last_became_peered":"2024-04-22T01:10:29.590035+0000","last_unstale":"2024-04-22T01:11:17.306511+0000","last_undegraded":"2024-04-22T01:11:17.306511+0000","last_fullsized":"2024-04-22T01:11:17.306511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:33.546044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308053+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:11:17.308053+0000","last_peered":"2024-04-22T01:11:17.308053+0000","last_clean":"2024-04-22T01:11:17.308053+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:11:17.308053+0000","last_undegraded":"2024-04-22T01:11:17.308053+0000","last_fullsized":"2024-04-22T01:11:17.308053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310804+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:11:17.310804+0000","last_peered":"2024-04-22T01:11:17.310804+0000","last_clean":"2024-04-22T01:11:17.310804+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:11:17.310804+0000","last_undegraded":"2024-04-22T01:11:17.310804+0000","last_fullsized":"2024-04-22T01:11:17.310804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:10.538440+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:12:10.538440+0000","last_peered":"2024-04-22T01:12:10.538440+0000","last_clean":"2024-04-22T01:12:10.538440+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:12:10.538440+0000","last_undegraded":"2024-04-22T01:12:10.538440+0000","last_fullsized":"2024-04-22T01:12:10.538440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307528+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:11:17.307528+0000","last_peered":"2024-04-22T01:11:17.307528+0000","last_clean":"2024-04-22T01:11:17.307528+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:11:17.307528+0000","last_undegraded":"2024-04-22T01:11:17.307528+0000","last_fullsized":"2024-04-22T01:11:17.307528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310994+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:11:17.310994+0000","last_peered":"2024-04-22T01:11:17.310994+0000","last_clean":"2024-04-22T01:11:17.310994+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:11:17.310994+0000","last_undegraded":"2024-04-22T01:11:17.310994+0000","last_fullsized":"2024-04-22T01:11:17.310994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586033+0000","last_change":"2024-04-22T01:11:17.586033+0000","last_active":"2024-04-22T01:11:17.586033+0000","last_peered":"2024-04-22T01:11:17.586033+0000","last_clean":"2024-04-22T01:11:17.586033+0000","last_became_active":"2024-04-22T01:11:17.585529+0000","last_became_peered":"2024-04-22T01:11:17.585529+0000","last_unstale":"2024-04-22T01:11:17.586033+0000","last_undegraded":"2024-04-22T01:11:17.586033+0000","last_fullsized":"2024-04-22T01:11:17.586033+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:35:49.219761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:11.565059+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:12:11.565059+0000","last_peered":"2024-04-22T01:12:11.565059+0000","last_clean":"2024-04-22T01:12:11.565059+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:12:11.565059+0000","last_undegraded":"2024-04-22T01:12:11.565059+0000","last_fullsized":"2024-04-22T01:12:11.565059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312107+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:11:17.312107+0000","last_peered":"2024-04-22T01:11:17.312107+0000","last_clean":"2024-04-22T01:11:17.312107+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:11:17.312107+0000","last_undegraded":"2024-04-22T01:11:17.312107+0000","last_fullsized":"2024-04-22T01:11:17.312107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523430+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:11:17.523430+0000","last_peered":"2024-04-22T01:11:17.523430+0000","last_clean":"2024-04-22T01:11:17.523430+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:11:17.523430+0000","last_undegraded":"2024-04-22T01:11:17.523430+0000","last_fullsized":"2024-04-22T01:11:17.523430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305482+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:11:17.305482+0000","last_peered":"2024-04-22T01:11:17.305482+0000","last_clean":"2024-04-22T01:11:17.305482+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:11:17.305482+0000","last_undegraded":"2024-04-22T01:11:17.305482+0000","last_fullsized":"2024-04-22T01:11:17.305482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312239+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:11:17.312239+0000","last_peered":"2024-04-22T01:11:17.312239+0000","last_clean":"2024-04-22T01:11:17.312239+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:11:17.312239+0000","last_undegraded":"2024-04-22T01:11:17.312239+0000","last_fullsized":"2024-04-22T01:11:17.312239+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307944+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:11:17.307944+0000","last_peered":"2024-04-22T01:11:17.307944+0000","last_clean":"2024-04-22T01:11:17.307944+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:11:17.307944+0000","last_undegraded":"2024-04-22T01:11:17.307944+0000","last_fullsized":"2024-04-22T01:11:17.307944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310214+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:11:17.310214+0000","last_peered":"2024-04-22T01:11:17.310214+0000","last_clean":"2024-04-22T01:11:17.310214+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:11:17.310214+0000","last_undegraded":"2024-04-22T01:11:17.310214+0000","last_fullsized":"2024-04-22T01:11:17.310214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312293+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:11:17.312293+0000","last_peered":"2024-04-22T01:11:17.312293+0000","last_clean":"2024-04-22T01:11:17.312293+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:11:17.312293+0000","last_undegraded":"2024-04-22T01:11:17.312293+0000","last_fullsized":"2024-04-22T01:11:17.312293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836908+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:11:17.836908+0000","last_peered":"2024-04-22T01:11:17.836908+0000","last_clean":"2024-04-22T01:11:17.836908+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:11:17.836908+0000","last_undegraded":"2024-04-22T01:11:17.836908+0000","last_fullsized":"2024-04-22T01:11:17.836908+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584772+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:11:17.584772+0000","last_peered":"2024-04-22T01:11:17.584772+0000","last_clean":"2024-04-22T01:11:17.584772+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:11:17.584772+0000","last_undegraded":"2024-04-22T01:11:17.584772+0000","last_fullsized":"2024-04-22T01:11:17.584772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311092+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:11:17.311092+0000","last_peered":"2024-04-22T01:11:17.311092+0000","last_clean":"2024-04-22T01:11:17.311092+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:11:17.311092+0000","last_undegraded":"2024-04-22T01:11:17.311092+0000","last_fullsized":"2024-04-22T01:11:17.311092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.522935+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:11:17.522935+0000","last_peered":"2024-04-22T01:11:17.522935+0000","last_clean":"2024-04-22T01:11:17.522935+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:11:17.522935+0000","last_undegraded":"2024-04-22T01:11:17.522935+0000","last_fullsized":"2024-04-22T01:11:17.522935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'11","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307079+0000","last_change":"2024-04-22T01:10:29.598225+0000","last_active":"2024-04-22T01:11:17.307079+0000","last_peered":"2024-04-22T01:11:17.307079+0000","last_clean":"2024-04-22T01:11:17.307079+0000","last_became_active":"2024-04-22T01:10:29.597867+0000","last_became_peered":"2024-04-22T01:10:29.597867+0000","last_unstale":"2024-04-22T01:11:17.307079+0000","last_undegraded":"2024-04-22T01:11:17.307079+0000","last_fullsized":"2024-04-22T01:11:17.307079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:52:32.759350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523968+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:11:17.523968+0000","last_peered":"2024-04-22T01:11:17.523968+0000","last_clean":"2024-04-22T01:11:17.523968+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:11:17.523968+0000","last_undegraded":"2024-04-22T01:11:17.523968+0000","last_fullsized":"2024-04-22T01:11:17.523968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304704+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:11:17.304704+0000","last_peered":"2024-04-22T01:11:17.304704+0000","last_clean":"2024-04-22T01:11:17.304704+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:11:17.304704+0000","last_undegraded":"2024-04-22T01:11:17.304704+0000","last_fullsized":"2024-04-22T01:11:17.304704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306270+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:11:17.306270+0000","last_peered":"2024-04-22T01:11:17.306270+0000","last_clean":"2024-04-22T01:11:17.306270+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:11:17.306270+0000","last_undegraded":"2024-04-22T01:11:17.306270+0000","last_fullsized":"2024-04-22T01:11:17.306270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.524897+0000","last_change":"2024-04-22T01:11:17.524897+0000","last_active":"2024-04-22T01:11:17.524897+0000","last_peered":"2024-04-22T01:11:17.524897+0000","last_clean":"2024-04-22T01:11:17.524897+0000","last_became_active":"2024-04-22T01:11:17.524423+0000","last_became_peered":"2024-04-22T01:11:17.524423+0000","last_unstale":"2024-04-22T01:11:17.524897+0000","last_undegraded":"2024-04-22T01:11:17.524897+0000","last_fullsized":"2024-04-22T01:11:17.524897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:15.247122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305386+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:11:17.305386+0000","last_peered":"2024-04-22T01:11:17.305386+0000","last_clean":"2024-04-22T01:11:17.305386+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:11:17.305386+0000","last_undegraded":"2024-04-22T01:11:17.305386+0000","last_fullsized":"2024-04-22T01:11:17.305386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308542+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:11:17.308542+0000","last_peered":"2024-04-22T01:11:17.308542+0000","last_clean":"2024-04-22T01:11:17.308542+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:11:17.308542+0000","last_undegraded":"2024-04-22T01:11:17.308542+0000","last_fullsized":"2024-04-22T01:11:17.308542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584239+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:11:17.584239+0000","last_peered":"2024-04-22T01:11:17.584239+0000","last_clean":"2024-04-22T01:11:17.584239+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:11:17.584239+0000","last_undegraded":"2024-04-22T01:11:17.584239+0000","last_fullsized":"2024-04-22T01:11:17.584239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.525008+0000","last_change":"2024-04-22T01:11:17.525008+0000","last_active":"2024-04-22T01:11:17.525008+0000","last_peered":"2024-04-22T01:11:17.525008+0000","last_clean":"2024-04-22T01:11:17.525008+0000","last_became_active":"2024-04-22T01:11:17.524640+0000","last_became_peered":"2024-04-22T01:11:17.524640+0000","last_unstale":"2024-04-22T01:11:17.525008+0000","last_undegraded":"2024-04-22T01:11:17.525008+0000","last_fullsized":"2024-04-22T01:11:17.525008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:54.573851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523157+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:11:17.523157+0000","last_peered":"2024-04-22T01:11:17.523157+0000","last_clean":"2024-04-22T01:11:17.523157+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:11:17.523157+0000","last_undegraded":"2024-04-22T01:11:17.523157+0000","last_fullsized":"2024-04-22T01:11:17.523157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836711+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:11:17.836711+0000","last_peered":"2024-04-22T01:11:17.836711+0000","last_clean":"2024-04-22T01:11:17.836711+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:11:17.836711+0000","last_undegraded":"2024-04-22T01:11:17.836711+0000","last_fullsized":"2024-04-22T01:11:17.836711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305485+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:11:17.305485+0000","last_peered":"2024-04-22T01:11:17.305485+0000","last_clean":"2024-04-22T01:11:17.305485+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:11:17.305485+0000","last_undegraded":"2024-04-22T01:11:17.305485+0000","last_fullsized":"2024-04-22T01:11:17.305485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304832+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:11:17.304832+0000","last_peered":"2024-04-22T01:11:17.304832+0000","last_clean":"2024-04-22T01:11:17.304832+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:11:17.304832+0000","last_undegraded":"2024-04-22T01:11:17.304832+0000","last_fullsized":"2024-04-22T01:11:17.304832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523449+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:11:17.523449+0000","last_peered":"2024-04-22T01:11:17.523449+0000","last_clean":"2024-04-22T01:11:17.523449+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:11:17.523449+0000","last_undegraded":"2024-04-22T01:11:17.523449+0000","last_fullsized":"2024-04-22T01:11:17.523449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305334+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:11:17.305334+0000","last_peered":"2024-04-22T01:11:17.305334+0000","last_clean":"2024-04-22T01:11:17.305334+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:11:17.305334+0000","last_undegraded":"2024-04-22T01:11:17.305334+0000","last_fullsized":"2024-04-22T01:11:17.305334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584325+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:11:17.584325+0000","last_peered":"2024-04-22T01:11:17.584325+0000","last_clean":"2024-04-22T01:11:17.584325+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:11:17.584325+0000","last_undegraded":"2024-04-22T01:11:17.584325+0000","last_fullsized":"2024-04-22T01:11:17.584325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311417+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:11:17.311417+0000","last_peered":"2024-04-22T01:11:17.311417+0000","last_clean":"2024-04-22T01:11:17.311417+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:11:17.311417+0000","last_undegraded":"2024-04-22T01:11:17.311417+0000","last_fullsized":"2024-04-22T01:11:17.311417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.309497+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:11:17.309497+0000","last_peered":"2024-04-22T01:11:17.309497+0000","last_clean":"2024-04-22T01:11:17.309497+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:11:17.309497+0000","last_undegraded":"2024-04-22T01:11:17.309497+0000","last_fullsized":"2024-04-22T01:11:17.309497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:07.534956+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:12:07.534956+0000","last_peered":"2024-04-22T01:12:07.534956+0000","last_clean":"2024-04-22T01:12:07.534956+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:12:07.534956+0000","last_undegraded":"2024-04-22T01:12:07.534956+0000","last_fullsized":"2024-04-22T01:12:07.534956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312562+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:11:17.312562+0000","last_peered":"2024-04-22T01:11:17.312562+0000","last_clean":"2024-04-22T01:11:17.312562+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:11:17.312562+0000","last_undegraded":"2024-04-22T01:11:17.312562+0000","last_fullsized":"2024-04-22T01:11:17.312562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308327+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:11:17.308327+0000","last_peered":"2024-04-22T01:11:17.308327+0000","last_clean":"2024-04-22T01:11:17.308327+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:11:17.308327+0000","last_undegraded":"2024-04-22T01:11:17.308327+0000","last_fullsized":"2024-04-22T01:11:17.308327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308436+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:11:17.308436+0000","last_peered":"2024-04-22T01:11:17.308436+0000","last_clean":"2024-04-22T01:11:17.308436+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:11:17.308436+0000","last_undegraded":"2024-04-22T01:11:17.308436+0000","last_fullsized":"2024-04-22T01:11:17.308436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.313900+0000","last_change":"2024-04-22T01:11:17.313900+0000","last_active":"2024-04-22T01:11:17.313900+0000","last_peered":"2024-04-22T01:11:17.313900+0000","last_clean":"2024-04-22T01:11:17.313900+0000","last_became_active":"2024-04-22T01:11:17.313493+0000","last_became_peered":"2024-04-22T01:11:17.313493+0000","last_unstale":"2024-04-22T01:11:17.313900+0000","last_undegraded":"2024-04-22T01:11:17.313900+0000","last_fullsized":"2024-04-22T01:11:17.313900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:30:42.909373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307510+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:11:17.307510+0000","last_peered":"2024-04-22T01:11:17.307510+0000","last_clean":"2024-04-22T01:11:17.307510+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:11:17.307510+0000","last_undegraded":"2024-04-22T01:11:17.307510+0000","last_fullsized":"2024-04-22T01:11:17.307510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310993+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:11:17.310993+0000","last_peered":"2024-04-22T01:11:17.310993+0000","last_clean":"2024-04-22T01:11:17.310993+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:11:17.310993+0000","last_undegraded":"2024-04-22T01:11:17.310993+0000","last_fullsized":"2024-04-22T01:11:17.310993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312793+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:11:17.312793+0000","last_peered":"2024-04-22T01:11:17.312793+0000","last_clean":"2024-04-22T01:11:17.312793+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:11:17.312793+0000","last_undegraded":"2024-04-22T01:11:17.312793+0000","last_fullsized":"2024-04-22T01:11:17.312793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836801+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:11:17.836801+0000","last_peered":"2024-04-22T01:11:17.836801+0000","last_clean":"2024-04-22T01:11:17.836801+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:11:17.836801+0000","last_undegraded":"2024-04-22T01:11:17.836801+0000","last_fullsized":"2024-04-22T01:11:17.836801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304882+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:11:17.304882+0000","last_peered":"2024-04-22T01:11:17.304882+0000","last_clean":"2024-04-22T01:11:17.304882+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:11:17.304882+0000","last_undegraded":"2024-04-22T01:11:17.304882+0000","last_fullsized":"2024-04-22T01:11:17.304882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304899+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:11:17.304899+0000","last_peered":"2024-04-22T01:11:17.304899+0000","last_clean":"2024-04-22T01:11:17.304899+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:11:17.304899+0000","last_undegraded":"2024-04-22T01:11:17.304899+0000","last_fullsized":"2024-04-22T01:11:17.304899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308707+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:11:17.308707+0000","last_peered":"2024-04-22T01:11:17.308707+0000","last_clean":"2024-04-22T01:11:17.308707+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:11:17.308707+0000","last_undegraded":"2024-04-22T01:11:17.308707+0000","last_fullsized":"2024-04-22T01:11:17.308707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310948+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:11:17.310948+0000","last_peered":"2024-04-22T01:11:17.310948+0000","last_clean":"2024-04-22T01:11:17.310948+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:11:17.310948+0000","last_undegraded":"2024-04-22T01:11:17.310948+0000","last_fullsized":"2024-04-22T01:11:17.310948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837226+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:11:17.837226+0000","last_peered":"2024-04-22T01:11:17.837226+0000","last_clean":"2024-04-22T01:11:17.837226+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:11:17.837226+0000","last_undegraded":"2024-04-22T01:11:17.837226+0000","last_fullsized":"2024-04-22T01:11:17.837226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305757+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:11:17.305757+0000","last_peered":"2024-04-22T01:11:17.305757+0000","last_clean":"2024-04-22T01:11:17.305757+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:11:17.305757+0000","last_undegraded":"2024-04-22T01:11:17.305757+0000","last_fullsized":"2024-04-22T01:11:17.305757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.524212+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:11:17.524212+0000","last_peered":"2024-04-22T01:11:17.524212+0000","last_clean":"2024-04-22T01:11:17.524212+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:11:17.524212+0000","last_undegraded":"2024-04-22T01:11:17.524212+0000","last_fullsized":"2024-04-22T01:11:17.524212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307807+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:11:17.307807+0000","last_peered":"2024-04-22T01:11:17.307807+0000","last_clean":"2024-04-22T01:11:17.307807+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:11:17.307807+0000","last_undegraded":"2024-04-22T01:11:17.307807+0000","last_fullsized":"2024-04-22T01:11:17.307807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523359+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:11:17.523359+0000","last_peered":"2024-04-22T01:11:17.523359+0000","last_clean":"2024-04-22T01:11:17.523359+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:11:17.523359+0000","last_undegraded":"2024-04-22T01:11:17.523359+0000","last_fullsized":"2024-04-22T01:11:17.523359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586313+0000","last_change":"2024-04-22T01:11:17.586313+0000","last_active":"2024-04-22T01:11:17.586313+0000","last_peered":"2024-04-22T01:11:17.586313+0000","last_clean":"2024-04-22T01:11:17.586313+0000","last_became_active":"2024-04-22T01:11:17.585705+0000","last_became_peered":"2024-04-22T01:11:17.585705+0000","last_unstale":"2024-04-22T01:11:17.586313+0000","last_undegraded":"2024-04-22T01:11:17.586313+0000","last_fullsized":"2024-04-22T01:11:17.586313+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:02.313774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310499+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:11:17.310499+0000","last_peered":"2024-04-22T01:11:17.310499+0000","last_clean":"2024-04-22T01:11:17.310499+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:11:17.310499+0000","last_undegraded":"2024-04-22T01:11:17.310499+0000","last_fullsized":"2024-04-22T01:11:17.310499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837026+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:11:17.837026+0000","last_peered":"2024-04-22T01:11:17.837026+0000","last_clean":"2024-04-22T01:11:17.837026+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:11:17.837026+0000","last_undegraded":"2024-04-22T01:11:17.837026+0000","last_fullsized":"2024-04-22T01:11:17.837026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67846,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":750,"num_read_kb":452,"num_write":238,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501832,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1148899,"data_compressed":57044,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":505467,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":501749,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1098246,"data_compressed":57044,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":496875,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":558547,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":498722,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":559549,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":496583,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":500949,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1093828,"data_compressed":57044,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:12:14.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T01:12:14.903 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:15.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:14 smithi111 ceph-mon[31057]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:15.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:14 smithi111 ceph-mon[31057]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:12:15.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:14 smithi038 ceph-mon[28043]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:15.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:14 smithi038 ceph-mon[28043]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:12:15.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:14 smithi203 ceph-mon[31825]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:15.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:14 smithi203 ceph-mon[31825]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:12:16.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-22T01:12:16.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-22T01:12:16.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-22T01:12:16.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-22T01:12:16.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:16 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10, 5, 8]}]: dispatch 2024-04-22T01:12:16.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 -- 172.21.15.38:0/404830418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30102ec0 msgr2=0x7f1a30106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:16.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 --2- 172.21.15.38:0/404830418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30102ec0 0x7f1a30106eb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009990 tx=0x7f1a20009ca0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 -- 172.21.15.38:0/404830418 shutdown_connections 2024-04-22T01:12:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 --2- 172.21.15.38:0/404830418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a30107590 0x7f1a3010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 --2- 172.21.15.38:0/404830418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 0x7f1a301028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 --2- 172.21.15.38:0/404830418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30102ec0 0x7f1a30106eb0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 -- 172.21.15.38:0/404830418 >> 172.21.15.38:0/404830418 conn(0x7f1a300793a0 msgr2=0x7f1a300797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 -- 172.21.15.38:0/404830418 shutdown_connections 2024-04-22T01:12:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.678+0000 7f1a385d5700 1 -- 172.21.15.38:0/404830418 wait complete. 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 Processor -- start 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 -- start start 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 0x7f1a30074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a30102ec0 0x7f1a30074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 0x7f1a30074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1a30112530 con 0x7f1a30107590 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1a301123b0 con 0x7f1a30102510 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a385d5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1a301126b0 con 0x7f1a30102ec0 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 0x7f1a30074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36371700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 0x7f1a30074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 0x7f1a30074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52322/0 (socket says 172.21.15.38:52322) 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 -- 172.21.15.38:0/1919008679 learned_addr learned my addr 172.21.15.38:0/1919008679 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a35b70700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a30102ec0 0x7f1a30074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 -- 172.21.15.38:0/1919008679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a30102ec0 msgr2=0x7f1a30074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a30102ec0 0x7f1a30074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 -- 172.21.15.38:0/1919008679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 msgr2=0x7f1a30074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 0x7f1a30074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36b72700 1 -- 172.21.15.38:0/1919008679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a301a6f30 con 0x7f1a30107590 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.679+0000 7f1a36371700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 0x7f1a30074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.680+0000 7f1a36b72700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 0x7f1a30074fb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800ef30 tx=0x7f1a2800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.680+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a280092e0 con 0x7f1a30107590 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.680+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a28004bb0 con 0x7f1a30107590 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.680+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28004280 con 0x7f1a30107590 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.680+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a301a71c0 con 0x7f1a30107590 2024-04-22T01:12:16.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.680+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a301a7680 con 0x7f1a30107590 2024-04-22T01:12:16.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.681+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a30068f20 con 0x7f1a30107590 2024-04-22T01:12:16.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.682+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1a2801b030 con 0x7f1a30107590 2024-04-22T01:12:16.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.683+0000 7f1a277fe700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1a1c0a2c20 0x7f1a1c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:16.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.683+0000 7f1a36371700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1a1c0a2c20 0x7f1a1c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:16.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.683+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9216+0+0 (secure 0 0 0) 0x7f1a280ceea0 con 0x7f1a30107590 2024-04-22T01:12:16.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.683+0000 7f1a36371700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1a1c0a2c20 0x7f1a1c0a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c005950 tx=0x7f1a2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:16.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.687+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1a28022020 con 0x7f1a30107590 2024-04-22T01:12:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-22T01:12:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-22T01:12:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-22T01:12:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-22T01:12:16.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:16 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10, 5, 8]}]: dispatch 2024-04-22T01:12:16.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.888+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a30062670 con 0x7f1a1c0a2c20 2024-04-22T01:12:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-22T01:12:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-22T01:12:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-22T01:12:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-22T01:12:16.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:16 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10, 5, 8]}]: dispatch 2024-04-22T01:12:16.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.898+0000 7f1a277fe700 1 -- 172.21.15.38:0/1919008679 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340723 (secure 0 0 0) 0x7f1a30062670 con 0x7f1a1c0a2c20 2024-04-22T01:12:16.901 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1a1c0a2c20 msgr2=0x7f1a1c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1a1c0a2c20 0x7f1a1c0a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c005950 tx=0x7f1a2c00b410 comp rx=0 tx=0).stop 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 msgr2=0x7f1a30074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 0x7f1a30074fb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800ef30 tx=0x7f1a2800c3a0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 shutdown_connections 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1a1c0a2c20 0x7f1a1c0a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1a30107590 0x7f1a30074fb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a30102510 0x7f1a30074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 --2- 172.21.15.38:0/1919008679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a30102ec0 0x7f1a30074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:16.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 >> 172.21.15.38:0/1919008679 conn(0x7f1a300793a0 msgr2=0x7f1a30101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:16.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 shutdown_connections 2024-04-22T01:12:16.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:16.903+0000 7f1a385d5700 1 -- 172.21.15.38:0/1919008679 wait complete. 2024-04-22T01:12:16.905 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10, 5, 8]}]': finished 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-22T01:12:17.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:17 smithi038 ceph-mon[28043]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T01:12:17.708 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-04-22T01:12:16.463575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658448,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":896,"num_read_kb":566,"num_write":421,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":383,"ondisk_log_size":383,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357364,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584460,"statfs":{"total":1151940427776,"available":1151574487040,"internally_reserved":0,"allocated":12591104,"data_stored":7961246,"data_compressed":356811,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.654389"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837343+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:11:17.837343+0000","last_peered":"2024-04-22T01:11:17.837343+0000","last_clean":"2024-04-22T01:11:17.837343+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:11:17.837343+0000","last_undegraded":"2024-04-22T01:11:17.837343+0000","last_fullsized":"2024-04-22T01:11:17.837343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308435+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:11:17.308435+0000","last_peered":"2024-04-22T01:11:17.308435+0000","last_clean":"2024-04-22T01:11:17.308435+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:11:17.308435+0000","last_undegraded":"2024-04-22T01:11:17.308435+0000","last_fullsized":"2024-04-22T01:11:17.308435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308058+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:11:17.308058+0000","last_peered":"2024-04-22T01:11:17.308058+0000","last_clean":"2024-04-22T01:11:17.308058+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:11:17.308058+0000","last_undegraded":"2024-04-22T01:11:17.308058+0000","last_fullsized":"2024-04-22T01:11:17.308058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:15.804016+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:12:15.804016+0000","last_peered":"2024-04-22T01:12:15.804016+0000","last_clean":"2024-04-22T01:12:15.804016+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:12:15.804016+0000","last_undegraded":"2024-04-22T01:12:15.804016+0000","last_fullsized":"2024-04-22T01:12:15.804016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584434+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:11:17.584434+0000","last_peered":"2024-04-22T01:11:17.584434+0000","last_clean":"2024-04-22T01:11:17.584434+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:11:17.584434+0000","last_undegraded":"2024-04-22T01:11:17.584434+0000","last_fullsized":"2024-04-22T01:11:17.584434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307537+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:11:17.307537+0000","last_peered":"2024-04-22T01:11:17.307537+0000","last_clean":"2024-04-22T01:11:17.307537+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:11:17.307537+0000","last_undegraded":"2024-04-22T01:11:17.307537+0000","last_fullsized":"2024-04-22T01:11:17.307537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311342+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:11:17.311342+0000","last_peered":"2024-04-22T01:11:17.311342+0000","last_clean":"2024-04-22T01:11:17.311342+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:11:17.311342+0000","last_undegraded":"2024-04-22T01:11:17.311342+0000","last_fullsized":"2024-04-22T01:11:17.311342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.522657+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:11:17.522657+0000","last_peered":"2024-04-22T01:11:17.522657+0000","last_clean":"2024-04-22T01:11:17.522657+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:11:17.522657+0000","last_undegraded":"2024-04-22T01:11:17.522657+0000","last_fullsized":"2024-04-22T01:11:17.522657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310180+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:11:17.310180+0000","last_peered":"2024-04-22T01:11:17.310180+0000","last_clean":"2024-04-22T01:11:17.310180+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:11:17.310180+0000","last_undegraded":"2024-04-22T01:11:17.310180+0000","last_fullsized":"2024-04-22T01:11:17.310180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308084+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:11:17.308084+0000","last_peered":"2024-04-22T01:11:17.308084+0000","last_clean":"2024-04-22T01:11:17.308084+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:11:17.308084+0000","last_undegraded":"2024-04-22T01:11:17.308084+0000","last_fullsized":"2024-04-22T01:11:17.308084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312398+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:11:17.312398+0000","last_peered":"2024-04-22T01:11:17.312398+0000","last_clean":"2024-04-22T01:11:17.312398+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:11:17.312398+0000","last_undegraded":"2024-04-22T01:11:17.312398+0000","last_fullsized":"2024-04-22T01:11:17.312398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:15.803518+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:12:15.803518+0000","last_peered":"2024-04-22T01:12:15.803518+0000","last_clean":"2024-04-22T01:12:15.803518+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:12:15.803518+0000","last_undegraded":"2024-04-22T01:12:15.803518+0000","last_fullsized":"2024-04-22T01:12:15.803518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312524+0000","last_change":"2024-04-22T01:11:17.312524+0000","last_active":"2024-04-22T01:11:17.312524+0000","last_peered":"2024-04-22T01:11:17.312524+0000","last_clean":"2024-04-22T01:11:17.312524+0000","last_became_active":"2024-04-22T01:11:17.312068+0000","last_became_peered":"2024-04-22T01:11:17.312068+0000","last_unstale":"2024-04-22T01:11:17.312524+0000","last_undegraded":"2024-04-22T01:11:17.312524+0000","last_fullsized":"2024-04-22T01:11:17.312524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:47:17.243712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307962+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:11:17.307962+0000","last_peered":"2024-04-22T01:11:17.307962+0000","last_clean":"2024-04-22T01:11:17.307962+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:11:17.307962+0000","last_undegraded":"2024-04-22T01:11:17.307962+0000","last_fullsized":"2024-04-22T01:11:17.307962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308200+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:11:17.308200+0000","last_peered":"2024-04-22T01:11:17.308200+0000","last_clean":"2024-04-22T01:11:17.308200+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:11:17.308200+0000","last_undegraded":"2024-04-22T01:11:17.308200+0000","last_fullsized":"2024-04-22T01:11:17.308200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.309193+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:11:17.309193+0000","last_peered":"2024-04-22T01:11:17.309193+0000","last_clean":"2024-04-22T01:11:17.309193+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:11:17.309193+0000","last_undegraded":"2024-04-22T01:11:17.309193+0000","last_fullsized":"2024-04-22T01:11:17.309193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837232+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:11:17.837232+0000","last_peered":"2024-04-22T01:11:17.837232+0000","last_clean":"2024-04-22T01:11:17.837232+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:11:17.837232+0000","last_undegraded":"2024-04-22T01:11:17.837232+0000","last_fullsized":"2024-04-22T01:11:17.837232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:15.814867+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:12:15.814867+0000","last_peered":"2024-04-22T01:12:15.814867+0000","last_clean":"2024-04-22T01:12:15.814867+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:12:15.814867+0000","last_undegraded":"2024-04-22T01:12:15.814867+0000","last_fullsized":"2024-04-22T01:12:15.814867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305898+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:11:17.305898+0000","last_peered":"2024-04-22T01:11:17.305898+0000","last_clean":"2024-04-22T01:11:17.305898+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:11:17.305898+0000","last_undegraded":"2024-04-22T01:11:17.305898+0000","last_fullsized":"2024-04-22T01:11:17.305898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586121+0000","last_change":"2024-04-22T01:11:17.586121+0000","last_active":"2024-04-22T01:11:17.586121+0000","last_peered":"2024-04-22T01:11:17.586121+0000","last_clean":"2024-04-22T01:11:17.586121+0000","last_became_active":"2024-04-22T01:11:17.585619+0000","last_became_peered":"2024-04-22T01:11:17.585619+0000","last_unstale":"2024-04-22T01:11:17.586121+0000","last_undegraded":"2024-04-22T01:11:17.586121+0000","last_fullsized":"2024-04-22T01:11:17.586121+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:12.987810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:14.543271+0000","last_change":"2024-04-22T01:10:29.585069+0000","last_active":"2024-04-22T01:12:14.543271+0000","last_peered":"2024-04-22T01:12:14.543271+0000","last_clean":"2024-04-22T01:12:14.543271+0000","last_became_active":"2024-04-22T01:10:29.584651+0000","last_became_peered":"2024-04-22T01:10:29.584651+0000","last_unstale":"2024-04-22T01:12:14.543271+0000","last_undegraded":"2024-04-22T01:12:14.543271+0000","last_fullsized":"2024-04-22T01:12:14.543271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:20:13.121214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":77,"num_read_kb":77,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523254+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:11:17.523254+0000","last_peered":"2024-04-22T01:11:17.523254+0000","last_clean":"2024-04-22T01:11:17.523254+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:11:17.523254+0000","last_undegraded":"2024-04-22T01:11:17.523254+0000","last_fullsized":"2024-04-22T01:11:17.523254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305110+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:11:17.305110+0000","last_peered":"2024-04-22T01:11:17.305110+0000","last_clean":"2024-04-22T01:11:17.305110+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:11:17.305110+0000","last_undegraded":"2024-04-22T01:11:17.305110+0000","last_fullsized":"2024-04-22T01:11:17.305110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523715+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:11:17.523715+0000","last_peered":"2024-04-22T01:11:17.523715+0000","last_clean":"2024-04-22T01:11:17.523715+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:11:17.523715+0000","last_undegraded":"2024-04-22T01:11:17.523715+0000","last_fullsized":"2024-04-22T01:11:17.523715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308307+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:11:17.308307+0000","last_peered":"2024-04-22T01:11:17.308307+0000","last_clean":"2024-04-22T01:11:17.308307+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:11:17.308307+0000","last_undegraded":"2024-04-22T01:11:17.308307+0000","last_fullsized":"2024-04-22T01:11:17.308307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305419+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:11:17.305419+0000","last_peered":"2024-04-22T01:11:17.305419+0000","last_clean":"2024-04-22T01:11:17.305419+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:11:17.305419+0000","last_undegraded":"2024-04-22T01:11:17.305419+0000","last_fullsized":"2024-04-22T01:11:17.305419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584910+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:11:17.584910+0000","last_peered":"2024-04-22T01:11:17.584910+0000","last_clean":"2024-04-22T01:11:17.584910+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:11:17.584910+0000","last_undegraded":"2024-04-22T01:11:17.584910+0000","last_fullsized":"2024-04-22T01:11:17.584910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.309976+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:11:17.309976+0000","last_peered":"2024-04-22T01:11:17.309976+0000","last_clean":"2024-04-22T01:11:17.309976+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:11:17.309976+0000","last_undegraded":"2024-04-22T01:11:17.309976+0000","last_fullsized":"2024-04-22T01:11:17.309976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305145+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:11:17.305145+0000","last_peered":"2024-04-22T01:11:17.305145+0000","last_clean":"2024-04-22T01:11:17.305145+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:11:17.305145+0000","last_undegraded":"2024-04-22T01:11:17.305145+0000","last_fullsized":"2024-04-22T01:11:17.305145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584383+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:11:17.584383+0000","last_peered":"2024-04-22T01:11:17.584383+0000","last_clean":"2024-04-22T01:11:17.584383+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:11:17.584383+0000","last_undegraded":"2024-04-22T01:11:17.584383+0000","last_fullsized":"2024-04-22T01:11:17.584383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307891+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:11:17.307891+0000","last_peered":"2024-04-22T01:11:17.307891+0000","last_clean":"2024-04-22T01:11:17.307891+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:11:17.307891+0000","last_undegraded":"2024-04-22T01:11:17.307891+0000","last_fullsized":"2024-04-22T01:11:17.307891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305124+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:11:17.305124+0000","last_peered":"2024-04-22T01:11:17.305124+0000","last_clean":"2024-04-22T01:11:17.305124+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:11:17.305124+0000","last_undegraded":"2024-04-22T01:11:17.305124+0000","last_fullsized":"2024-04-22T01:11:17.305124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:13.543351+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:12:13.543351+0000","last_peered":"2024-04-22T01:12:13.543351+0000","last_clean":"2024-04-22T01:12:13.543351+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:12:13.543351+0000","last_undegraded":"2024-04-22T01:12:13.543351+0000","last_fullsized":"2024-04-22T01:12:13.543351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584145+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:11:17.584145+0000","last_peered":"2024-04-22T01:11:17.584145+0000","last_clean":"2024-04-22T01:11:17.584145+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:11:17.584145+0000","last_undegraded":"2024-04-22T01:11:17.584145+0000","last_fullsized":"2024-04-22T01:11:17.584145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305097+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:11:17.305097+0000","last_peered":"2024-04-22T01:11:17.305097+0000","last_clean":"2024-04-22T01:11:17.305097+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:11:17.305097+0000","last_undegraded":"2024-04-22T01:11:17.305097+0000","last_fullsized":"2024-04-22T01:11:17.305097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307542+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:11:17.307542+0000","last_peered":"2024-04-22T01:11:17.307542+0000","last_clean":"2024-04-22T01:11:17.307542+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:11:17.307542+0000","last_undegraded":"2024-04-22T01:11:17.307542+0000","last_fullsized":"2024-04-22T01:11:17.307542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307530+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:11:17.307530+0000","last_peered":"2024-04-22T01:11:17.307530+0000","last_clean":"2024-04-22T01:11:17.307530+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:11:17.307530+0000","last_undegraded":"2024-04-22T01:11:17.307530+0000","last_fullsized":"2024-04-22T01:11:17.307530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837202+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:11:17.837202+0000","last_peered":"2024-04-22T01:11:17.837202+0000","last_clean":"2024-04-22T01:11:17.837202+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:11:17.837202+0000","last_undegraded":"2024-04-22T01:11:17.837202+0000","last_fullsized":"2024-04-22T01:11:17.837202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310483+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:11:17.310483+0000","last_peered":"2024-04-22T01:11:17.310483+0000","last_clean":"2024-04-22T01:11:17.310483+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:11:17.310483+0000","last_undegraded":"2024-04-22T01:11:17.310483+0000","last_fullsized":"2024-04-22T01:11:17.310483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.522957+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:11:17.522957+0000","last_peered":"2024-04-22T01:11:17.522957+0000","last_clean":"2024-04-22T01:11:17.522957+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:11:17.522957+0000","last_undegraded":"2024-04-22T01:11:17.522957+0000","last_fullsized":"2024-04-22T01:11:17.522957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308289+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:11:17.308289+0000","last_peered":"2024-04-22T01:11:17.308289+0000","last_clean":"2024-04-22T01:11:17.308289+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:11:17.308289+0000","last_undegraded":"2024-04-22T01:11:17.308289+0000","last_fullsized":"2024-04-22T01:11:17.308289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307754+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:11:17.307754+0000","last_peered":"2024-04-22T01:11:17.307754+0000","last_clean":"2024-04-22T01:11:17.307754+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:11:17.307754+0000","last_undegraded":"2024-04-22T01:11:17.307754+0000","last_fullsized":"2024-04-22T01:11:17.307754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:15.814518+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:12:15.814518+0000","last_peered":"2024-04-22T01:12:15.814518+0000","last_clean":"2024-04-22T01:12:15.814518+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:12:15.814518+0000","last_undegraded":"2024-04-22T01:12:15.814518+0000","last_fullsized":"2024-04-22T01:12:15.814518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523060+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:11:17.523060+0000","last_peered":"2024-04-22T01:11:17.523060+0000","last_clean":"2024-04-22T01:11:17.523060+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:11:17.523060+0000","last_undegraded":"2024-04-22T01:11:17.523060+0000","last_fullsized":"2024-04-22T01:11:17.523060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:11.539982+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:12:11.539982+0000","last_peered":"2024-04-22T01:12:11.539982+0000","last_clean":"2024-04-22T01:12:11.539982+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:12:11.539982+0000","last_undegraded":"2024-04-22T01:12:11.539982+0000","last_fullsized":"2024-04-22T01:12:11.539982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310305+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:11:17.310305+0000","last_peered":"2024-04-22T01:11:17.310305+0000","last_clean":"2024-04-22T01:11:17.310305+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:11:17.310305+0000","last_undegraded":"2024-04-22T01:11:17.310305+0000","last_fullsized":"2024-04-22T01:11:17.310305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306083+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:11:17.306083+0000","last_peered":"2024-04-22T01:11:17.306083+0000","last_clean":"2024-04-22T01:11:17.306083+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:11:17.306083+0000","last_undegraded":"2024-04-22T01:11:17.306083+0000","last_fullsized":"2024-04-22T01:11:17.306083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312678+0000","last_change":"2024-04-22T01:10:29.591793+0000","last_active":"2024-04-22T01:11:17.312678+0000","last_peered":"2024-04-22T01:11:17.312678+0000","last_clean":"2024-04-22T01:11:17.312678+0000","last_became_active":"2024-04-22T01:10:29.591245+0000","last_became_peered":"2024-04-22T01:10:29.591245+0000","last_unstale":"2024-04-22T01:11:17.312678+0000","last_undegraded":"2024-04-22T01:11:17.312678+0000","last_fullsized":"2024-04-22T01:11:17.312678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:40:28.909648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308167+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:11:17.308167+0000","last_peered":"2024-04-22T01:11:17.308167+0000","last_clean":"2024-04-22T01:11:17.308167+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:11:17.308167+0000","last_undegraded":"2024-04-22T01:11:17.308167+0000","last_fullsized":"2024-04-22T01:11:17.308167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586221+0000","last_change":"2024-04-22T01:11:17.586221+0000","last_active":"2024-04-22T01:11:17.586221+0000","last_peered":"2024-04-22T01:11:17.586221+0000","last_clean":"2024-04-22T01:11:17.586221+0000","last_became_active":"2024-04-22T01:11:17.585689+0000","last_became_peered":"2024-04-22T01:11:17.585689+0000","last_unstale":"2024-04-22T01:11:17.586221+0000","last_undegraded":"2024-04-22T01:11:17.586221+0000","last_fullsized":"2024-04-22T01:11:17.586221+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:40.327272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307759+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:11:17.307759+0000","last_peered":"2024-04-22T01:11:17.307759+0000","last_clean":"2024-04-22T01:11:17.307759+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:11:17.307759+0000","last_undegraded":"2024-04-22T01:11:17.307759+0000","last_fullsized":"2024-04-22T01:11:17.307759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.585668+0000","last_change":"2024-04-22T01:11:17.585668+0000","last_active":"2024-04-22T01:11:17.585668+0000","last_peered":"2024-04-22T01:11:17.585668+0000","last_clean":"2024-04-22T01:11:17.585668+0000","last_became_active":"2024-04-22T01:11:17.585217+0000","last_became_peered":"2024-04-22T01:11:17.585217+0000","last_unstale":"2024-04-22T01:11:17.585668+0000","last_undegraded":"2024-04-22T01:11:17.585668+0000","last_fullsized":"2024-04-22T01:11:17.585668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:24.007210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306370+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:11:17.306370+0000","last_peered":"2024-04-22T01:11:17.306370+0000","last_clean":"2024-04-22T01:11:17.306370+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:11:17.306370+0000","last_undegraded":"2024-04-22T01:11:17.306370+0000","last_fullsized":"2024-04-22T01:11:17.306370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":209,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:11.546386+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:12:11.546386+0000","last_peered":"2024-04-22T01:12:11.546386+0000","last_clean":"2024-04-22T01:12:11.546386+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:12:11.546386+0000","last_undegraded":"2024-04-22T01:12:11.546386+0000","last_fullsized":"2024-04-22T01:12:11.546386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60642,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":150,"num_read_kb":134,"num_write":35,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584635+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:11:17.584635+0000","last_peered":"2024-04-22T01:11:17.584635+0000","last_clean":"2024-04-22T01:11:17.584635+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:11:17.584635+0000","last_undegraded":"2024-04-22T01:11:17.584635+0000","last_fullsized":"2024-04-22T01:11:17.584635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308625+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:11:17.308625+0000","last_peered":"2024-04-22T01:11:17.308625+0000","last_clean":"2024-04-22T01:11:17.308625+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:11:17.308625+0000","last_undegraded":"2024-04-22T01:11:17.308625+0000","last_fullsized":"2024-04-22T01:11:17.308625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305262+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:11:17.305262+0000","last_peered":"2024-04-22T01:11:17.305262+0000","last_clean":"2024-04-22T01:11:17.305262+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:11:17.305262+0000","last_undegraded":"2024-04-22T01:11:17.305262+0000","last_fullsized":"2024-04-22T01:11:17.305262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310549+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:11:17.310549+0000","last_peered":"2024-04-22T01:11:17.310549+0000","last_clean":"2024-04-22T01:11:17.310549+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:11:17.310549+0000","last_undegraded":"2024-04-22T01:11:17.310549+0000","last_fullsized":"2024-04-22T01:11:17.310549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836995+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:11:17.836995+0000","last_peered":"2024-04-22T01:11:17.836995+0000","last_clean":"2024-04-22T01:11:17.836995+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:11:17.836995+0000","last_undegraded":"2024-04-22T01:11:17.836995+0000","last_fullsized":"2024-04-22T01:11:17.836995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307337+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:11:17.307337+0000","last_peered":"2024-04-22T01:11:17.307337+0000","last_clean":"2024-04-22T01:11:17.307337+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:11:17.307337+0000","last_undegraded":"2024-04-22T01:11:17.307337+0000","last_fullsized":"2024-04-22T01:11:17.307337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311538+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:11:17.311538+0000","last_peered":"2024-04-22T01:11:17.311538+0000","last_clean":"2024-04-22T01:11:17.311538+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:11:17.311538+0000","last_undegraded":"2024-04-22T01:11:17.311538+0000","last_fullsized":"2024-04-22T01:11:17.311538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306262+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:11:17.306262+0000","last_peered":"2024-04-22T01:11:17.306262+0000","last_clean":"2024-04-22T01:11:17.306262+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:11:17.306262+0000","last_undegraded":"2024-04-22T01:11:17.306262+0000","last_fullsized":"2024-04-22T01:11:17.306262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:14.571273+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:12:14.571273+0000","last_peered":"2024-04-22T01:12:14.571273+0000","last_clean":"2024-04-22T01:12:14.571273+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:12:14.571273+0000","last_undegraded":"2024-04-22T01:12:14.571273+0000","last_fullsized":"2024-04-22T01:12:14.571273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.314098+0000","last_change":"2024-04-22T01:11:17.314098+0000","last_active":"2024-04-22T01:11:17.314098+0000","last_peered":"2024-04-22T01:11:17.314098+0000","last_clean":"2024-04-22T01:11:17.314098+0000","last_became_active":"2024-04-22T01:11:17.313663+0000","last_became_peered":"2024-04-22T01:11:17.313663+0000","last_unstale":"2024-04-22T01:11:17.314098+0000","last_undegraded":"2024-04-22T01:11:17.314098+0000","last_fullsized":"2024-04-22T01:11:17.314098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:20:03.826857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310992+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:11:17.310992+0000","last_peered":"2024-04-22T01:11:17.310992+0000","last_clean":"2024-04-22T01:11:17.310992+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:11:17.310992+0000","last_undegraded":"2024-04-22T01:11:17.310992+0000","last_fullsized":"2024-04-22T01:11:17.310992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584269+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:11:17.584269+0000","last_peered":"2024-04-22T01:11:17.584269+0000","last_clean":"2024-04-22T01:11:17.584269+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:11:17.584269+0000","last_undegraded":"2024-04-22T01:11:17.584269+0000","last_fullsized":"2024-04-22T01:11:17.584269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307005+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:11:17.307005+0000","last_peered":"2024-04-22T01:11:17.307005+0000","last_clean":"2024-04-22T01:11:17.307005+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:11:17.307005+0000","last_undegraded":"2024-04-22T01:11:17.307005+0000","last_fullsized":"2024-04-22T01:11:17.307005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307000+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:11:17.307000+0000","last_peered":"2024-04-22T01:11:17.307000+0000","last_clean":"2024-04-22T01:11:17.307000+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:11:17.307000+0000","last_undegraded":"2024-04-22T01:11:17.307000+0000","last_fullsized":"2024-04-22T01:11:17.307000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584776+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:11:17.584776+0000","last_peered":"2024-04-22T01:11:17.584776+0000","last_clean":"2024-04-22T01:11:17.584776+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:11:17.584776+0000","last_undegraded":"2024-04-22T01:11:17.584776+0000","last_fullsized":"2024-04-22T01:11:17.584776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837145+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:11:17.837145+0000","last_peered":"2024-04-22T01:11:17.837145+0000","last_clean":"2024-04-22T01:11:17.837145+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:11:17.837145+0000","last_undegraded":"2024-04-22T01:11:17.837145+0000","last_fullsized":"2024-04-22T01:11:17.837145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584360+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:11:17.584360+0000","last_peered":"2024-04-22T01:11:17.584360+0000","last_clean":"2024-04-22T01:11:17.584360+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:11:17.584360+0000","last_undegraded":"2024-04-22T01:11:17.584360+0000","last_fullsized":"2024-04-22T01:11:17.584360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306511+0000","last_change":"2024-04-22T01:10:29.590388+0000","last_active":"2024-04-22T01:11:17.306511+0000","last_peered":"2024-04-22T01:11:17.306511+0000","last_clean":"2024-04-22T01:11:17.306511+0000","last_became_active":"2024-04-22T01:10:29.590035+0000","last_became_peered":"2024-04-22T01:10:29.590035+0000","last_unstale":"2024-04-22T01:11:17.306511+0000","last_undegraded":"2024-04-22T01:11:17.306511+0000","last_fullsized":"2024-04-22T01:11:17.306511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:33.546044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308053+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:11:17.308053+0000","last_peered":"2024-04-22T01:11:17.308053+0000","last_clean":"2024-04-22T01:11:17.308053+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:11:17.308053+0000","last_undegraded":"2024-04-22T01:11:17.308053+0000","last_fullsized":"2024-04-22T01:11:17.308053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310804+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:11:17.310804+0000","last_peered":"2024-04-22T01:11:17.310804+0000","last_clean":"2024-04-22T01:11:17.310804+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:11:17.310804+0000","last_undegraded":"2024-04-22T01:11:17.310804+0000","last_fullsized":"2024-04-22T01:11:17.310804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:10.538440+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:12:10.538440+0000","last_peered":"2024-04-22T01:12:10.538440+0000","last_clean":"2024-04-22T01:12:10.538440+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:12:10.538440+0000","last_undegraded":"2024-04-22T01:12:10.538440+0000","last_fullsized":"2024-04-22T01:12:10.538440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307528+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:11:17.307528+0000","last_peered":"2024-04-22T01:11:17.307528+0000","last_clean":"2024-04-22T01:11:17.307528+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:11:17.307528+0000","last_undegraded":"2024-04-22T01:11:17.307528+0000","last_fullsized":"2024-04-22T01:11:17.307528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310994+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:11:17.310994+0000","last_peered":"2024-04-22T01:11:17.310994+0000","last_clean":"2024-04-22T01:11:17.310994+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:11:17.310994+0000","last_undegraded":"2024-04-22T01:11:17.310994+0000","last_fullsized":"2024-04-22T01:11:17.310994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586033+0000","last_change":"2024-04-22T01:11:17.586033+0000","last_active":"2024-04-22T01:11:17.586033+0000","last_peered":"2024-04-22T01:11:17.586033+0000","last_clean":"2024-04-22T01:11:17.586033+0000","last_became_active":"2024-04-22T01:11:17.585529+0000","last_became_peered":"2024-04-22T01:11:17.585529+0000","last_unstale":"2024-04-22T01:11:17.586033+0000","last_undegraded":"2024-04-22T01:11:17.586033+0000","last_fullsized":"2024-04-22T01:11:17.586033+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:35:49.219761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:11.565059+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:12:11.565059+0000","last_peered":"2024-04-22T01:12:11.565059+0000","last_clean":"2024-04-22T01:12:11.565059+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:12:11.565059+0000","last_undegraded":"2024-04-22T01:12:11.565059+0000","last_fullsized":"2024-04-22T01:12:11.565059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312107+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:11:17.312107+0000","last_peered":"2024-04-22T01:11:17.312107+0000","last_clean":"2024-04-22T01:11:17.312107+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:11:17.312107+0000","last_undegraded":"2024-04-22T01:11:17.312107+0000","last_fullsized":"2024-04-22T01:11:17.312107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523430+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:11:17.523430+0000","last_peered":"2024-04-22T01:11:17.523430+0000","last_clean":"2024-04-22T01:11:17.523430+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:11:17.523430+0000","last_undegraded":"2024-04-22T01:11:17.523430+0000","last_fullsized":"2024-04-22T01:11:17.523430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305482+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:11:17.305482+0000","last_peered":"2024-04-22T01:11:17.305482+0000","last_clean":"2024-04-22T01:11:17.305482+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:11:17.305482+0000","last_undegraded":"2024-04-22T01:11:17.305482+0000","last_fullsized":"2024-04-22T01:11:17.305482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312239+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:11:17.312239+0000","last_peered":"2024-04-22T01:11:17.312239+0000","last_clean":"2024-04-22T01:11:17.312239+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:11:17.312239+0000","last_undegraded":"2024-04-22T01:11:17.312239+0000","last_fullsized":"2024-04-22T01:11:17.312239+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307944+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:11:17.307944+0000","last_peered":"2024-04-22T01:11:17.307944+0000","last_clean":"2024-04-22T01:11:17.307944+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:11:17.307944+0000","last_undegraded":"2024-04-22T01:11:17.307944+0000","last_fullsized":"2024-04-22T01:11:17.307944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310214+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:11:17.310214+0000","last_peered":"2024-04-22T01:11:17.310214+0000","last_clean":"2024-04-22T01:11:17.310214+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:11:17.310214+0000","last_undegraded":"2024-04-22T01:11:17.310214+0000","last_fullsized":"2024-04-22T01:11:17.310214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312293+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:11:17.312293+0000","last_peered":"2024-04-22T01:11:17.312293+0000","last_clean":"2024-04-22T01:11:17.312293+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:11:17.312293+0000","last_undegraded":"2024-04-22T01:11:17.312293+0000","last_fullsized":"2024-04-22T01:11:17.312293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836908+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:11:17.836908+0000","last_peered":"2024-04-22T01:11:17.836908+0000","last_clean":"2024-04-22T01:11:17.836908+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:11:17.836908+0000","last_undegraded":"2024-04-22T01:11:17.836908+0000","last_fullsized":"2024-04-22T01:11:17.836908+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584772+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:11:17.584772+0000","last_peered":"2024-04-22T01:11:17.584772+0000","last_clean":"2024-04-22T01:11:17.584772+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:11:17.584772+0000","last_undegraded":"2024-04-22T01:11:17.584772+0000","last_fullsized":"2024-04-22T01:11:17.584772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311092+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:11:17.311092+0000","last_peered":"2024-04-22T01:11:17.311092+0000","last_clean":"2024-04-22T01:11:17.311092+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:11:17.311092+0000","last_undegraded":"2024-04-22T01:11:17.311092+0000","last_fullsized":"2024-04-22T01:11:17.311092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.522935+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:11:17.522935+0000","last_peered":"2024-04-22T01:11:17.522935+0000","last_clean":"2024-04-22T01:11:17.522935+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:11:17.522935+0000","last_undegraded":"2024-04-22T01:11:17.522935+0000","last_fullsized":"2024-04-22T01:11:17.522935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'11","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307079+0000","last_change":"2024-04-22T01:10:29.598225+0000","last_active":"2024-04-22T01:11:17.307079+0000","last_peered":"2024-04-22T01:11:17.307079+0000","last_clean":"2024-04-22T01:11:17.307079+0000","last_became_active":"2024-04-22T01:10:29.597867+0000","last_became_peered":"2024-04-22T01:10:29.597867+0000","last_unstale":"2024-04-22T01:11:17.307079+0000","last_undegraded":"2024-04-22T01:11:17.307079+0000","last_fullsized":"2024-04-22T01:11:17.307079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:52:32.759350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523968+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:11:17.523968+0000","last_peered":"2024-04-22T01:11:17.523968+0000","last_clean":"2024-04-22T01:11:17.523968+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:11:17.523968+0000","last_undegraded":"2024-04-22T01:11:17.523968+0000","last_fullsized":"2024-04-22T01:11:17.523968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304704+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:11:17.304704+0000","last_peered":"2024-04-22T01:11:17.304704+0000","last_clean":"2024-04-22T01:11:17.304704+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:11:17.304704+0000","last_undegraded":"2024-04-22T01:11:17.304704+0000","last_fullsized":"2024-04-22T01:11:17.304704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.306270+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:11:17.306270+0000","last_peered":"2024-04-22T01:11:17.306270+0000","last_clean":"2024-04-22T01:11:17.306270+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:11:17.306270+0000","last_undegraded":"2024-04-22T01:11:17.306270+0000","last_fullsized":"2024-04-22T01:11:17.306270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.524897+0000","last_change":"2024-04-22T01:11:17.524897+0000","last_active":"2024-04-22T01:11:17.524897+0000","last_peered":"2024-04-22T01:11:17.524897+0000","last_clean":"2024-04-22T01:11:17.524897+0000","last_became_active":"2024-04-22T01:11:17.524423+0000","last_became_peered":"2024-04-22T01:11:17.524423+0000","last_unstale":"2024-04-22T01:11:17.524897+0000","last_undegraded":"2024-04-22T01:11:17.524897+0000","last_fullsized":"2024-04-22T01:11:17.524897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:15.247122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305386+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:11:17.305386+0000","last_peered":"2024-04-22T01:11:17.305386+0000","last_clean":"2024-04-22T01:11:17.305386+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:11:17.305386+0000","last_undegraded":"2024-04-22T01:11:17.305386+0000","last_fullsized":"2024-04-22T01:11:17.305386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308542+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:11:17.308542+0000","last_peered":"2024-04-22T01:11:17.308542+0000","last_clean":"2024-04-22T01:11:17.308542+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:11:17.308542+0000","last_undegraded":"2024-04-22T01:11:17.308542+0000","last_fullsized":"2024-04-22T01:11:17.308542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584239+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:11:17.584239+0000","last_peered":"2024-04-22T01:11:17.584239+0000","last_clean":"2024-04-22T01:11:17.584239+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:11:17.584239+0000","last_undegraded":"2024-04-22T01:11:17.584239+0000","last_fullsized":"2024-04-22T01:11:17.584239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.525008+0000","last_change":"2024-04-22T01:11:17.525008+0000","last_active":"2024-04-22T01:11:17.525008+0000","last_peered":"2024-04-22T01:11:17.525008+0000","last_clean":"2024-04-22T01:11:17.525008+0000","last_became_active":"2024-04-22T01:11:17.524640+0000","last_became_peered":"2024-04-22T01:11:17.524640+0000","last_unstale":"2024-04-22T01:11:17.525008+0000","last_undegraded":"2024-04-22T01:11:17.525008+0000","last_fullsized":"2024-04-22T01:11:17.525008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:54.573851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523157+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:11:17.523157+0000","last_peered":"2024-04-22T01:11:17.523157+0000","last_clean":"2024-04-22T01:11:17.523157+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:11:17.523157+0000","last_undegraded":"2024-04-22T01:11:17.523157+0000","last_fullsized":"2024-04-22T01:11:17.523157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836711+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:11:17.836711+0000","last_peered":"2024-04-22T01:11:17.836711+0000","last_clean":"2024-04-22T01:11:17.836711+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:11:17.836711+0000","last_undegraded":"2024-04-22T01:11:17.836711+0000","last_fullsized":"2024-04-22T01:11:17.836711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305485+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:11:17.305485+0000","last_peered":"2024-04-22T01:11:17.305485+0000","last_clean":"2024-04-22T01:11:17.305485+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:11:17.305485+0000","last_undegraded":"2024-04-22T01:11:17.305485+0000","last_fullsized":"2024-04-22T01:11:17.305485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304832+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:11:17.304832+0000","last_peered":"2024-04-22T01:11:17.304832+0000","last_clean":"2024-04-22T01:11:17.304832+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:11:17.304832+0000","last_undegraded":"2024-04-22T01:11:17.304832+0000","last_fullsized":"2024-04-22T01:11:17.304832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523449+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:11:17.523449+0000","last_peered":"2024-04-22T01:11:17.523449+0000","last_clean":"2024-04-22T01:11:17.523449+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:11:17.523449+0000","last_undegraded":"2024-04-22T01:11:17.523449+0000","last_fullsized":"2024-04-22T01:11:17.523449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305334+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:11:17.305334+0000","last_peered":"2024-04-22T01:11:17.305334+0000","last_clean":"2024-04-22T01:11:17.305334+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:11:17.305334+0000","last_undegraded":"2024-04-22T01:11:17.305334+0000","last_fullsized":"2024-04-22T01:11:17.305334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.584325+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:11:17.584325+0000","last_peered":"2024-04-22T01:11:17.584325+0000","last_clean":"2024-04-22T01:11:17.584325+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:11:17.584325+0000","last_undegraded":"2024-04-22T01:11:17.584325+0000","last_fullsized":"2024-04-22T01:11:17.584325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.311417+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:11:17.311417+0000","last_peered":"2024-04-22T01:11:17.311417+0000","last_clean":"2024-04-22T01:11:17.311417+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:11:17.311417+0000","last_undegraded":"2024-04-22T01:11:17.311417+0000","last_fullsized":"2024-04-22T01:11:17.311417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.309497+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:11:17.309497+0000","last_peered":"2024-04-22T01:11:17.309497+0000","last_clean":"2024-04-22T01:11:17.309497+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:11:17.309497+0000","last_undegraded":"2024-04-22T01:11:17.309497+0000","last_fullsized":"2024-04-22T01:11:17.309497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:12:12.541903+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:12:12.541903+0000","last_peered":"2024-04-22T01:12:12.541903+0000","last_clean":"2024-04-22T01:12:12.541903+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:12:12.541903+0000","last_undegraded":"2024-04-22T01:12:12.541903+0000","last_fullsized":"2024-04-22T01:12:12.541903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312562+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:11:17.312562+0000","last_peered":"2024-04-22T01:11:17.312562+0000","last_clean":"2024-04-22T01:11:17.312562+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:11:17.312562+0000","last_undegraded":"2024-04-22T01:11:17.312562+0000","last_fullsized":"2024-04-22T01:11:17.312562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308327+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:11:17.308327+0000","last_peered":"2024-04-22T01:11:17.308327+0000","last_clean":"2024-04-22T01:11:17.308327+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:11:17.308327+0000","last_undegraded":"2024-04-22T01:11:17.308327+0000","last_fullsized":"2024-04-22T01:11:17.308327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308436+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:11:17.308436+0000","last_peered":"2024-04-22T01:11:17.308436+0000","last_clean":"2024-04-22T01:11:17.308436+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:11:17.308436+0000","last_undegraded":"2024-04-22T01:11:17.308436+0000","last_fullsized":"2024-04-22T01:11:17.308436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.313900+0000","last_change":"2024-04-22T01:11:17.313900+0000","last_active":"2024-04-22T01:11:17.313900+0000","last_peered":"2024-04-22T01:11:17.313900+0000","last_clean":"2024-04-22T01:11:17.313900+0000","last_became_active":"2024-04-22T01:11:17.313493+0000","last_became_peered":"2024-04-22T01:11:17.313493+0000","last_unstale":"2024-04-22T01:11:17.313900+0000","last_undegraded":"2024-04-22T01:11:17.313900+0000","last_fullsized":"2024-04-22T01:11:17.313900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:30:42.909373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307510+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:11:17.307510+0000","last_peered":"2024-04-22T01:11:17.307510+0000","last_clean":"2024-04-22T01:11:17.307510+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:11:17.307510+0000","last_undegraded":"2024-04-22T01:11:17.307510+0000","last_fullsized":"2024-04-22T01:11:17.307510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310993+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:11:17.310993+0000","last_peered":"2024-04-22T01:11:17.310993+0000","last_clean":"2024-04-22T01:11:17.310993+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:11:17.310993+0000","last_undegraded":"2024-04-22T01:11:17.310993+0000","last_fullsized":"2024-04-22T01:11:17.310993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.312793+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:11:17.312793+0000","last_peered":"2024-04-22T01:11:17.312793+0000","last_clean":"2024-04-22T01:11:17.312793+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:11:17.312793+0000","last_undegraded":"2024-04-22T01:11:17.312793+0000","last_fullsized":"2024-04-22T01:11:17.312793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.836801+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:11:17.836801+0000","last_peered":"2024-04-22T01:11:17.836801+0000","last_clean":"2024-04-22T01:11:17.836801+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:11:17.836801+0000","last_undegraded":"2024-04-22T01:11:17.836801+0000","last_fullsized":"2024-04-22T01:11:17.836801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304882+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:11:17.304882+0000","last_peered":"2024-04-22T01:11:17.304882+0000","last_clean":"2024-04-22T01:11:17.304882+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:11:17.304882+0000","last_undegraded":"2024-04-22T01:11:17.304882+0000","last_fullsized":"2024-04-22T01:11:17.304882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.304899+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:11:17.304899+0000","last_peered":"2024-04-22T01:11:17.304899+0000","last_clean":"2024-04-22T01:11:17.304899+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:11:17.304899+0000","last_undegraded":"2024-04-22T01:11:17.304899+0000","last_fullsized":"2024-04-22T01:11:17.304899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.308707+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:11:17.308707+0000","last_peered":"2024-04-22T01:11:17.308707+0000","last_clean":"2024-04-22T01:11:17.308707+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:11:17.308707+0000","last_undegraded":"2024-04-22T01:11:17.308707+0000","last_fullsized":"2024-04-22T01:11:17.308707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310948+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:11:17.310948+0000","last_peered":"2024-04-22T01:11:17.310948+0000","last_clean":"2024-04-22T01:11:17.310948+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:11:17.310948+0000","last_undegraded":"2024-04-22T01:11:17.310948+0000","last_fullsized":"2024-04-22T01:11:17.310948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837226+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:11:17.837226+0000","last_peered":"2024-04-22T01:11:17.837226+0000","last_clean":"2024-04-22T01:11:17.837226+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:11:17.837226+0000","last_undegraded":"2024-04-22T01:11:17.837226+0000","last_fullsized":"2024-04-22T01:11:17.837226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.305757+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:11:17.305757+0000","last_peered":"2024-04-22T01:11:17.305757+0000","last_clean":"2024-04-22T01:11:17.305757+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:11:17.305757+0000","last_undegraded":"2024-04-22T01:11:17.305757+0000","last_fullsized":"2024-04-22T01:11:17.305757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.524212+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:11:17.524212+0000","last_peered":"2024-04-22T01:11:17.524212+0000","last_clean":"2024-04-22T01:11:17.524212+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:11:17.524212+0000","last_undegraded":"2024-04-22T01:11:17.524212+0000","last_fullsized":"2024-04-22T01:11:17.524212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.307807+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:11:17.307807+0000","last_peered":"2024-04-22T01:11:17.307807+0000","last_clean":"2024-04-22T01:11:17.307807+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:11:17.307807+0000","last_undegraded":"2024-04-22T01:11:17.307807+0000","last_fullsized":"2024-04-22T01:11:17.307807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.523359+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:11:17.523359+0000","last_peered":"2024-04-22T01:11:17.523359+0000","last_clean":"2024-04-22T01:11:17.523359+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:11:17.523359+0000","last_undegraded":"2024-04-22T01:11:17.523359+0000","last_fullsized":"2024-04-22T01:11:17.523359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.586313+0000","last_change":"2024-04-22T01:11:17.586313+0000","last_active":"2024-04-22T01:11:17.586313+0000","last_peered":"2024-04-22T01:11:17.586313+0000","last_clean":"2024-04-22T01:11:17.586313+0000","last_became_active":"2024-04-22T01:11:17.585705+0000","last_became_peered":"2024-04-22T01:11:17.585705+0000","last_unstale":"2024-04-22T01:11:17.586313+0000","last_undegraded":"2024-04-22T01:11:17.586313+0000","last_fullsized":"2024-04-22T01:11:17.586313+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:02.313774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.310499+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:11:17.310499+0000","last_peered":"2024-04-22T01:11:17.310499+0000","last_clean":"2024-04-22T01:11:17.310499+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:11:17.310499+0000","last_undegraded":"2024-04-22T01:11:17.310499+0000","last_fullsized":"2024-04-22T01:11:17.310499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T01:11:17.837026+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:11:17.837026+0000","last_peered":"2024-04-22T01:11:17.837026+0000","last_clean":"2024-04-22T01:11:17.837026+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:11:17.837026+0000","last_undegraded":"2024-04-22T01:11:17.837026+0000","last_fullsized":"2024-04-22T01:11:17.837026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67846,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":770,"num_read_kb":457,"num_write":238,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383715,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501832,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1148899,"data_compressed":57044,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":505467,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":501749,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1098246,"data_compressed":57044,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":496875,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":558547,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":498722,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":559549,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":496583,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":500949,"data_compressed":20631,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1093828,"data_compressed":57044,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:12:17.712 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T01:12:17.713 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T01:12:17.713 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T01:12:17.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10, 5, 8]}]': finished 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-22T01:12:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:17 smithi203 ceph-mon[31825]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10, 5, 8]}]': finished 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T01:12:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-22T01:12:17.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:17 smithi111 ceph-mon[31057]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T01:12:18.001 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:18.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:18 smithi203 ceph-mon[31825]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:12:18.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:18 smithi038 ceph-mon[28043]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:12:18.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:18 smithi111 ceph-mon[31057]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:12:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:19 smithi038 ceph-mon[28043]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-22T01:12:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:19 smithi038 ceph-mon[28043]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T01:12:19.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:19 smithi203 ceph-mon[31825]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-22T01:12:19.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:19 smithi203 ceph-mon[31825]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 -- 172.21.15.38:0/2921045655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 msgr2=0x7f9104105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 --2- 172.21.15.38:0/2921045655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f9104105d90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0099c0 tx=0x7f90ec009cd0 comp rx=0 tx=0).stop 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 -- 172.21.15.38:0/2921045655 shutdown_connections 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 --2- 172.21.15.38:0/2921045655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 0x7f910410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 --2- 172.21.15.38:0/2921045655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104101420 0x7f9104101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 --2- 172.21.15.38:0/2921045655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f9104105d90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 -- 172.21.15.38:0/2921045655 >> 172.21.15.38:0/2921045655 conn(0x7f9104077180 msgr2=0x7f9104077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 -- 172.21.15.38:0/2921045655 shutdown_connections 2024-04-22T01:12:19.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.813+0000 7f910a545700 1 -- 172.21.15.38:0/2921045655 wait complete. 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.814+0000 7f910a545700 1 Processor -- start 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.814+0000 7f910a545700 1 -- start start 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f910a545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104101420 0x7f910419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f910a545700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f910419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f910a545700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 0x7f9104197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f910a545700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9104111450 con 0x7f9104101dd0 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f910a545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91041112d0 con 0x7f9104101420 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f910a545700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91041115d0 con 0x7f9104106470 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f910419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f910419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39242/0 (socket says 172.21.15.38:39242) 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 -- 172.21.15.38:0/532220097 learned_addr learned my addr 172.21.15.38:0/532220097 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f9103fff700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104101420 0x7f910419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f9108ae2700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 0x7f9104197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 -- 172.21.15.38:0/532220097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 msgr2=0x7f9104197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:19.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 0x7f9104197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 -- 172.21.15.38:0/532220097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104101420 msgr2=0x7f910419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104101420 0x7f910419d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 -- 172.21.15.38:0/532220097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9104198010 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f9108ae2700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 0x7f9104197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.815+0000 7f91037fe700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f910419dba0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00fa00 tx=0x7f90ec00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.816+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec027070 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.816+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91041982a0 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.816+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90ec003680 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.816+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec0173f0 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.816+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91041a45e0 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.817+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9104102870 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.818+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f90ec02a070 con 0x7f9104101dd0 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.818+0000 7f91017fa700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90f00a2c20 0x7f90f00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:19.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.819+0000 7f9103fff700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90f00a2c20 0x7f90f00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:19.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.819+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f90ec0cffc0 con 0x7f9104101dd0 2024-04-22T01:12:19.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.819+0000 7f9103fff700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90f00a2c20 0x7f90f00a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009c60 tx=0x7f90f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:19.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:19.823+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f90ec013070 con 0x7f9104101dd0 2024-04-22T01:12:19.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:19 smithi111 ceph-mon[31057]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 639 B/s wr, 12 op/s 2024-04-22T01:12:19.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:19 smithi111 ceph-mon[31057]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T01:12:20.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.078+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9104068f20 con 0x7f9104101dd0 2024-04-22T01:12:20.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.079+0000 7f91017fa700 1 -- 172.21.15.38:0/532220097 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+181 (secure 0 0 0) 0x7f90ec09cbe0 con 0x7f9104101dd0 2024-04-22T01:12:20.078 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:20.078 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs peering","count":2},"muted":false}},"mutes":[]} 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.081+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90f00a2c20 msgr2=0x7f90f00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.081+0000 7f910a545700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90f00a2c20 0x7f90f00a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009c60 tx=0x7f90f4009260 comp rx=0 tx=0).stop 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.081+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 msgr2=0x7f910419dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.081+0000 7f910a545700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f910419dba0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00fa00 tx=0x7f90ec00fa30 comp rx=0 tx=0).stop 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 shutdown_connections 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f90f00a2c20 0x7f90f00a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9104106470 0x7f9104197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104101420 0x7f910419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 --2- 172.21.15.38:0/532220097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9104101dd0 0x7f910419dba0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 >> 172.21.15.38:0/532220097 conn(0x7f9104077180 msgr2=0x7f91040ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:20.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 shutdown_connections 2024-04-22T01:12:20.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:20.082+0000 7f910a545700 1 -- 172.21.15.38:0/532220097 wait complete. 2024-04-22T01:12:20.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:20 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/532220097' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T01:12:20.782 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:20 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/532220097' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T01:12:20.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:20 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/532220097' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T01:12:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:21 smithi203 ceph-mon[31825]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-22T01:12:21.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:21 smithi111 ceph-mon[31057]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-22T01:12:21.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:21 smithi038 ceph-mon[28043]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-22T01:12:23.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:23 smithi203 ceph-mon[31825]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 895 B/s wr, 12 op/s; 83 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:23.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T01:12:23.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:23 smithi038 ceph-mon[28043]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 895 B/s wr, 12 op/s; 83 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:23.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:23 smithi111 ceph-mon[31057]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 895 B/s wr, 12 op/s; 83 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:24.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:24 smithi111 ceph-mon[31057]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T01:12:24.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:24 smithi111 ceph-mon[31057]: Cluster is now healthy 2024-04-22T01:12:24.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:24 smithi038 ceph-mon[28043]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T01:12:24.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:24 smithi038 ceph-mon[28043]: Cluster is now healthy 2024-04-22T01:12:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:24 smithi203 ceph-mon[31825]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T01:12:25.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:24 smithi203 ceph-mon[31825]: Cluster is now healthy 2024-04-22T01:12:25.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:25 smithi038 ceph-mon[28043]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 895 B/s wr, 12 op/s; 83 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:25 smithi111 ceph-mon[31057]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 895 B/s wr, 12 op/s; 83 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1962470473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a98101af0 msgr2=0x7f5a9810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1962470473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a98101af0 0x7f5a9810dfe0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000b0a0 tx=0x7f5a9000b3b0 comp rx=0 tx=0).stop 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1962470473 shutdown_connections 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1962470473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a98101af0 0x7f5a9810dfe0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1962470473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a981071a0 0x7f5a98107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1962470473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a98101130 0x7f5a981015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1962470473 >> 172.21.15.38:0/1962470473 conn(0x7f5a980780e0 msgr2=0x7f5a980784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1962470473 shutdown_connections 2024-04-22T01:12:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.933+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1962470473 wait complete. 2024-04-22T01:12:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.934+0000 7f5a9ca27700 1 Processor -- start 2024-04-22T01:12:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.934+0000 7f5a9ca27700 1 -- start start 2024-04-22T01:12:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9ca27700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a98101130 0x7f5a9819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9ca27700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a98101af0 0x7f5a9819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9ca27700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 0x7f5a98198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9ca27700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5a98111410 con 0x7f5a981071a0 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9ca27700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5a98111290 con 0x7f5a98101af0 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9ca27700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a98111590 con 0x7f5a98101130 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 0x7f5a98198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 0x7f5a98198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39258/0 (socket says 172.21.15.38:39258) 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 -- 172.21.15.38:0/1059463291 learned_addr learned my addr 172.21.15.38:0/1059463291 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a95d9b700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a98101af0 0x7f5a9819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a9659c700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a98101130 0x7f5a9819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 -- 172.21.15.38:0/1059463291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a98101130 msgr2=0x7f5a9819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a98101130 0x7f5a9819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 -- 172.21.15.38:0/1059463291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a98101af0 msgr2=0x7f5a9819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a98101af0 0x7f5a9819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:25.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 -- 172.21.15.38:0/1059463291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a98199060 con 0x7f5a981071a0 2024-04-22T01:12:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.935+0000 7f5a96d9d700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 0x7f5a98198780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5a900062a0 tx=0x7f5a90004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.936+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90021070 con 0x7f5a981071a0 2024-04-22T01:12:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.936+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a90007c00 con 0x7f5a981071a0 2024-04-22T01:12:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.936+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a981992f0 con 0x7f5a981071a0 2024-04-22T01:12:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.936+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9001d450 con 0x7f5a981071a0 2024-04-22T01:12:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.936+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a981a55e0 con 0x7f5a981071a0 2024-04-22T01:12:25.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.937+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a98102850 con 0x7f5a981071a0 2024-04-22T01:12:25.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.938+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5a90004020 con 0x7f5a981071a0 2024-04-22T01:12:25.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.938+0000 7f5a8f7fe700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5a840a2ca0 0x7f5a840a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:25.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.939+0000 7f5a9659c700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5a840a2ca0 0x7f5a840a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:25.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.939+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f5a9000f020 con 0x7f5a981071a0 2024-04-22T01:12:25.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.939+0000 7f5a9659c700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5a840a2ca0 0x7f5a840a5160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a80005950 tx=0x7f5a800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:25.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:25.943+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5a9002a020 con 0x7f5a981071a0 2024-04-22T01:12:26.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:25 smithi203 ceph-mon[31825]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 895 B/s wr, 12 op/s; 83 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:26.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.195+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5a981a57b0 con 0x7f5a981071a0 2024-04-22T01:12:26.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.195+0000 7f5a8f7fe700 1 -- 172.21.15.38:0/1059463291 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5a9009d550 con 0x7f5a981071a0 2024-04-22T01:12:26.195 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:26.196 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T01:12:26.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5a840a2ca0 msgr2=0x7f5a840a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5a840a2ca0 0x7f5a840a5160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a80005950 tx=0x7f5a800058e0 comp rx=0 tx=0).stop 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 msgr2=0x7f5a98198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 0x7f5a98198780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5a900062a0 tx=0x7f5a90004d70 comp rx=0 tx=0).stop 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 shutdown_connections 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f5a840a2ca0 0x7f5a840a5160 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a981071a0 0x7f5a98198780 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a98101130 0x7f5a9819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 --2- 172.21.15.38:0/1059463291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a98101af0 0x7f5a9819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 >> 172.21.15.38:0/1059463291 conn(0x7f5a980780e0 msgr2=0x7f5a98100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 shutdown_connections 2024-04-22T01:12:26.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:26.197+0000 7f5a9ca27700 1 -- 172.21.15.38:0/1059463291 wait complete. 2024-04-22T01:12:26.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:26 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1059463291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T01:12:26.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:26 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1059463291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T01:12:26.999 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T01:12:26.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:12:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:26 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1059463291' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T01:12:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 -- 172.21.15.38:0/2091380580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 msgr2=0x7f37181020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 --2- 172.21.15.38:0/2091380580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f37181020a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3708009cd0 tx=0x7f370802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 -- 172.21.15.38:0/2091380580 shutdown_connections 2024-04-22T01:12:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 --2- 172.21.15.38:0/2091380580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37181012e0 0x7f3718108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 --2- 172.21.15.38:0/2091380580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3718100920 0x7f3718100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 --2- 172.21.15.38:0/2091380580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f37181020a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 -- 172.21.15.38:0/2091380580 >> 172.21.15.38:0/2091380580 conn(0x7f37180fb3b0 msgr2=0x7f37180fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 -- 172.21.15.38:0/2091380580 shutdown_connections 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 -- 172.21.15.38:0/2091380580 wait complete. 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 Processor -- start 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.145+0000 7f371d266640 1 -- start start 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3718100920 0x7f3718197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37181012e0 0x7f3718198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f371819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f371810c410 con 0x7f3718101c80 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f371810c290 con 0x7f3718100920 2024-04-22T01:12:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f371810c590 con 0x7f37181012e0 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f371819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f371819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49386/0 (socket says 172.21.15.38:49386) 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 -- 172.21.15.38:0/3161363979 learned_addr learned my addr 172.21.15.38:0/3161363979 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37167fc640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37181012e0 0x7f3718198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f3716ffd640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3718100920 0x7f3718197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 -- 172.21.15.38:0/3161363979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37181012e0 msgr2=0x7f3718198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37181012e0 0x7f3718198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 -- 172.21.15.38:0/3161363979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3718100920 msgr2=0x7f3718197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3718100920 0x7f3718197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 -- 172.21.15.38:0/3161363979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371819f800 con 0x7f3718101c80 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f37177fe640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f371819f190 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3710009fd0 tx=0x7f371000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3710015020 con 0x7f3718101c80 2024-04-22T01:12:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3710011d10 con 0x7f3718101c80 2024-04-22T01:12:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.146+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371819faf0 con 0x7f3718101c80 2024-04-22T01:12:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.147+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371000f2e0 con 0x7f3718101c80 2024-04-22T01:12:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.147+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371819fff0 con 0x7f3718101c80 2024-04-22T01:12:27.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.147+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37181a02e0 con 0x7f3718101c80 2024-04-22T01:12:27.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.149+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3710018020 con 0x7f3718101c80 2024-04-22T01:12:27.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.149+0000 7f36fffff640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37000a2b90 0x7f37000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.149+0000 7f3716ffd640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37000a2b90 0x7f37000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.149+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f37100cece0 con 0x7f3718101c80 2024-04-22T01:12:27.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.150+0000 7f3716ffd640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37000a2b90 0x7f37000a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f370802c700 tx=0x7f370802cca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:27.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.154+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f371001f020 con 0x7f3718101c80 2024-04-22T01:12:27.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.339+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37181a04b0 con 0x7f3718101c80 2024-04-22T01:12:27.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.341+0000 7f36fffff640 1 -- 172.21.15.38:0/3161363979 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7f371009b870 con 0x7f3718101c80 2024-04-22T01:12:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:27.342 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.343+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37000a2b90 msgr2=0x7f37000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.343+0000 7f371d266640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37000a2b90 0x7f37000a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f370802c700 tx=0x7f370802cca0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.343+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 msgr2=0x7f371819f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.343+0000 7f371d266640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f371819f190 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3710009fd0 tx=0x7f371000b820 comp rx=0 tx=0).stop 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.343+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 shutdown_connections 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f37000a2b90 0x7f37000a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3718101c80 0x7f371819f190 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37181012e0 0x7f3718198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 --2- 172.21.15.38:0/3161363979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3718100920 0x7f3718197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 >> 172.21.15.38:0/3161363979 conn(0x7f37180fb3b0 msgr2=0x7f37180fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:27.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 shutdown_connections 2024-04-22T01:12:27.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.344+0000 7f371d266640 1 -- 172.21.15.38:0/3161363979 wait complete. 2024-04-22T01:12:27.356 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:12:27.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.461+0000 7fa22d484640 1 -- 172.21.15.38:0/392070101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228100680 msgr2=0x7fa228100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.461+0000 7fa22d484640 1 --2- 172.21.15.38:0/392070101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228100680 0x7fa228100aa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009900 tx=0x7fa21c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 -- 172.21.15.38:0/392070101 shutdown_connections 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 --2- 172.21.15.38:0/392070101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 0x7fa228108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 --2- 172.21.15.38:0/392070101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228101870 0x7fa228101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 --2- 172.21.15.38:0/392070101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228100680 0x7fa228100aa0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 -- 172.21.15.38:0/392070101 >> 172.21.15.38:0/392070101 conn(0x7fa2280fbbf0 msgr2=0x7fa2280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:27.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 -- 172.21.15.38:0/392070101 shutdown_connections 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 -- 172.21.15.38:0/392070101 wait complete. 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 Processor -- start 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.462+0000 7fa22d484640 1 -- start start 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228100680 0x7fa22810ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 0x7fa2281094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 0x7fa228109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa22810d3e0 con 0x7fa228101870 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa22810d260 con 0x7fa228100680 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa22810d560 con 0x7fa228102230 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 0x7fa2281094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 0x7fa2281094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49406/0 (socket says 172.21.15.38:49406) 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 -- 172.21.15.38:0/2649004615 learned_addr learned my addr 172.21.15.38:0/2649004615 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:27.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa226ffd640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228100680 0x7fa22810ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2277fe640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 0x7fa228109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 -- 172.21.15.38:0/2649004615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 msgr2=0x7fa228109a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 0x7fa228109a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 -- 172.21.15.38:0/2649004615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228100680 msgr2=0x7fa22810ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228100680 0x7fa22810ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 -- 172.21.15.38:0/2649004615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22810a290 con 0x7fa228101870 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa226ffd640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228100680 0x7fa22810ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2277fe640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 0x7fa228109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa2267fc640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 0x7fa2281094c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e9f0 tx=0x7fa21800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21800cd80 con 0x7fa228101870 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2281a1c60 con 0x7fa228101870 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa218010910 con 0x7fa228101870 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa218018950 con 0x7fa228101870 2024-04-22T01:12:27.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.463+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2281a21b0 con 0x7fa228101870 2024-04-22T01:12:27.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.464+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa228107ac0 con 0x7fa228101870 2024-04-22T01:12:27.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.466+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa218010ac0 con 0x7fa228101870 2024-04-22T01:12:27.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.466+0000 7fa20ffff640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2100a2b90 0x7fa2100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.466+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa2180cf930 con 0x7fa228101870 2024-04-22T01:12:27.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.466+0000 7fa226ffd640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2100a2b90 0x7fa2100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.467+0000 7fa226ffd640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2100a2b90 0x7fa2100a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa21c02c7b0 tx=0x7fa21c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:27.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.471+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa21809c4c0 con 0x7fa228101870 2024-04-22T01:12:27.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:27 smithi038 ceph-mon[28043]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 766 B/s wr, 10 op/s; 71 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:27.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:27 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3161363979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:27.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.658+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2280623e0 con 0x7fa228101870 2024-04-22T01:12:27.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.659+0000 7fa20ffff640 1 -- 172.21.15.38:0/2649004615 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fa21809bdc0 con 0x7fa228101870 2024-04-22T01:12:27.659 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:12:27.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2100a2b90 msgr2=0x7fa2100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2100a2b90 0x7fa2100a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa21c02c7b0 tx=0x7fa21c005fd0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 msgr2=0x7fa2281094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 0x7fa2281094c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e9f0 tx=0x7fa21800eeb0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 shutdown_connections 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2100a2b90 0x7fa2100a5050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa228102230 0x7fa228109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa228101870 0x7fa2281094c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 --2- 172.21.15.38:0/2649004615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa228100680 0x7fa22810ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 >> 172.21.15.38:0/2649004615 conn(0x7fa2280fbbf0 msgr2=0x7fa228104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 shutdown_connections 2024-04-22T01:12:27.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.661+0000 7fa22d484640 1 -- 172.21.15.38:0/2649004615 wait complete. 2024-04-22T01:12:27.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:12:27.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.776+0000 7fb0fce6d640 1 -- 172.21.15.38:0/1576561429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f8102250 msgr2=0x7fb0f8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.776+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/1576561429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f8102250 0x7fb0f8108ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009900 tx=0x7fb0f002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 -- 172.21.15.38:0/1576561429 shutdown_connections 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/1576561429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f8102250 0x7fb0f8108ae0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/1576561429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f8101890 0x7fb0f8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/1576561429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f81006a0 0x7fb0f8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 -- 172.21.15.38:0/1576561429 >> 172.21.15.38:0/1576561429 conn(0x7fb0f80fbc30 msgr2=0x7fb0f80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 -- 172.21.15.38:0/1576561429 shutdown_connections 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 -- 172.21.15.38:0/1576561429 wait complete. 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 Processor -- start 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.777+0000 7fb0fce6d640 1 -- start start 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 0x7fb0f81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f8101890 0x7fb0f8196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f8102250 0x7fb0f819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f810c390 con 0x7fb0f81006a0 2024-04-22T01:12:27.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f810c210 con 0x7fb0f8102250 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f810c510 con 0x7fb0f8101890 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 0x7fb0f81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 0x7fb0f81969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49430/0 (socket says 172.21.15.38:49430) 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 -- 172.21.15.38:0/2117794835 learned_addr learned my addr 172.21.15.38:0/2117794835 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f6d9d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f8102250 0x7fb0f819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f5d9b640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f8101890 0x7fb0f8196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 -- 172.21.15.38:0/2117794835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f8101890 msgr2=0x7fb0f8196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f8101890 0x7fb0f8196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 -- 172.21.15.38:0/2117794835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f8102250 msgr2=0x7fb0f819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f8102250 0x7fb0f819df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 -- 172.21.15.38:0/2117794835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f819e5d0 con 0x7fb0f81006a0 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f5d9b640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f8101890 0x7fb0f8196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0f659c640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 0x7fb0f81969b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800b9c0 tx=0x7fb0e800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:27.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8019070 con 0x7fb0f81006a0 2024-04-22T01:12:27.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0e800cba0 con 0x7fb0f81006a0 2024-04-22T01:12:27.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f819e8c0 con 0x7fb0f81006a0 2024-04-22T01:12:27.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8012640 con 0x7fb0f81006a0 2024-04-22T01:12:27.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.778+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f819ee10 con 0x7fb0f81006a0 2024-04-22T01:12:27.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.780+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f81079c0 con 0x7fb0f81006a0 2024-04-22T01:12:27.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.781+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb0e801c070 con 0x7fb0f81006a0 2024-04-22T01:12:27.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.781+0000 7fb0ef7fe640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb0e40a2b90 0x7fb0e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:27.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.781+0000 7fb0f5d9b640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb0e40a2b90 0x7fb0e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:27.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.781+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fb0e80ce9c0 con 0x7fb0f81006a0 2024-04-22T01:12:27.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.782+0000 7fb0f5d9b640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb0e40a2b90 0x7fb0e40a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb0e00058c0 tx=0x7fb0e0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:27.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.786+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb0e800cd50 con 0x7fb0f81006a0 2024-04-22T01:12:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:27 smithi111 ceph-mon[31057]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 766 B/s wr, 10 op/s; 71 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:27 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3161363979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:27.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:27 smithi203 ceph-mon[31825]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 766 B/s wr, 10 op/s; 71 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:27.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:27 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3161363979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:27.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.972+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb0f8062340 con 0x7fb0f81006a0 2024-04-22T01:12:27.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.973+0000 7fb0ef7fe640 1 -- 172.21.15.38:0/2117794835 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fb0e809af00 con 0x7fb0f81006a0 2024-04-22T01:12:27.973 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:27.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb0e40a2b90 msgr2=0x7fb0e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb0e40a2b90 0x7fb0e40a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb0e00058c0 tx=0x7fb0e0005850 comp rx=0 tx=0).stop 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 msgr2=0x7fb0f81969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 0x7fb0f81969b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800b9c0 tx=0x7fb0e800be80 comp rx=0 tx=0).stop 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 shutdown_connections 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fb0e40a2b90 0x7fb0e40a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f8102250 0x7fb0f819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0f8101890 0x7fb0f8196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 --2- 172.21.15.38:0/2117794835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0f81006a0 0x7fb0f81969b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 >> 172.21.15.38:0/2117794835 conn(0x7fb0f80fbc30 msgr2=0x7fb0f8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 shutdown_connections 2024-04-22T01:12:27.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:27.975+0000 7fb0fce6d640 1 -- 172.21.15.38:0/2117794835 wait complete. 2024-04-22T01:12:27.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:12:28.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 -- 172.21.15.38:0/2894060487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 msgr2=0x7f22d4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/2894060487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d4100ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009900 tx=0x7f22d002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 -- 172.21.15.38:0/2894060487 shutdown_connections 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/2894060487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 0x7f22d4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/2894060487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d4101890 0x7f22d4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/2894060487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d4100ac0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 -- 172.21.15.38:0/2894060487 >> 172.21.15.38:0/2894060487 conn(0x7f22d40fbc30 msgr2=0x7f22d40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 -- 172.21.15.38:0/2894060487 shutdown_connections 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.091+0000 7f22dc4d4640 1 -- 172.21.15.38:0/2894060487 wait complete. 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 Processor -- start 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 -- start start 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d4101890 0x7f22d419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 0x7f22d41a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410c2f0 con 0x7f22d41006a0 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410c170 con 0x7f22d4102250 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22dc4d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410c470 con 0x7f22d4101890 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d406a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49456/0 (socket says 172.21.15.38:49456) 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 -- 172.21.15.38:0/901709548 learned_addr learned my addr 172.21.15.38:0/901709548 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 -- 172.21.15.38:0/901709548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d4101890 msgr2=0x7f22d419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22d9a6f640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d4101890 0x7f22d419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22daa71640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 0x7f22d41a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d4101890 0x7f22d419ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 -- 172.21.15.38:0/901709548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 msgr2=0x7f22d41a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 0x7f22d41a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22da270640 1 -- 172.21.15.38:0/901709548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d41a18f0 con 0x7f22d41006a0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.092+0000 7f22daa71640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 0x7f22d41a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.093+0000 7f22da270640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d406a8b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000c00 tx=0x7f22d0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.093+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0041070 con 0x7f22d41006a0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.093+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d41a1b20 con 0x7f22d41006a0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.093+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d00054e0 con 0x7f22d41006a0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.093+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d00346a0 con 0x7f22d41006a0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.093+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d41a2010 con 0x7f22d41006a0 2024-04-22T01:12:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.094+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d41079c0 con 0x7f22d41006a0 2024-04-22T01:12:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.096+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f22d0044070 con 0x7f22d41006a0 2024-04-22T01:12:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.096+0000 7f22cb7fe640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22c00a2bc0 0x7f22c00a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.096+0000 7f22d9a6f640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22c00a2bc0 0x7f22c00a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.096+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f22d00f13f0 con 0x7f22d41006a0 2024-04-22T01:12:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.097+0000 7f22d9a6f640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22c00a2bc0 0x7f22c00a5080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f22d406a620 tx=0x7f22c4005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:28.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.100+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f22d00bdf80 con 0x7f22d41006a0 2024-04-22T01:12:28.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.284+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22d402d3e0 con 0x7f22d41006a0 2024-04-22T01:12:28.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.285+0000 7f22cb7fe640 1 -- 172.21.15.38:0/901709548 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f22d0003e20 con 0x7f22d41006a0 2024-04-22T01:12:28.285 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:12:28.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.286+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22c00a2bc0 msgr2=0x7f22c00a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.286+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22c00a2bc0 0x7f22c00a5080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f22d406a620 tx=0x7f22c4005cc0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.286+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 msgr2=0x7f22d406a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.286+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d406a8b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000c00 tx=0x7f22d0004040 comp rx=0 tx=0).stop 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 shutdown_connections 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f22c00a2bc0 0x7f22c00a5080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22d4102250 0x7f22d41a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d4101890 0x7f22d419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 --2- 172.21.15.38:0/901709548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22d41006a0 0x7f22d406a8b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 >> 172.21.15.38:0/901709548 conn(0x7f22d40fbc30 msgr2=0x7f22d4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 shutdown_connections 2024-04-22T01:12:28.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.287+0000 7f22dc4d4640 1 -- 172.21.15.38:0/901709548 wait complete. 2024-04-22T01:12:28.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:12:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 -- 172.21.15.38:0/3924583906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090107460 msgr2=0x7f00901098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 --2- 172.21.15.38:0/3924583906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090107460 0x7f00901098b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0088009900 tx=0x7f008802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 -- 172.21.15.38:0/3924583906 shutdown_connections 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 --2- 172.21.15.38:0/3924583906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090107460 0x7f00901098b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 --2- 172.21.15.38:0/3924583906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 0x7f0090106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 --2- 172.21.15.38:0/3924583906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090102200 0x7f00901045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 -- 172.21.15.38:0/3924583906 >> 172.21.15.38:0/3924583906 conn(0x7f00900fbbf0 msgr2=0x7f00900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 -- 172.21.15.38:0/3924583906 shutdown_connections 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.403+0000 7f0095808640 1 -- 172.21.15.38:0/3924583906 wait complete. 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 Processor -- start 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 -- start start 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 0x7f0090101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 0x7f00901001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090107460 0x7f00901006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f009010d4f0 con 0x7f0090102200 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f009010d370 con 0x7f0090104b30 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f0095808640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f009010d670 con 0x7f0090107460 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 0x7f0090101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 0x7f0090101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49470/0 (socket says 172.21.15.38:49470) 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 -- 172.21.15.38:0/3141582728 learned_addr learned my addr 172.21.15.38:0/3141582728 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008f7fe640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090107460 0x7f00901006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008e7fc640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 0x7f00901001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 -- 172.21.15.38:0/3141582728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090107460 msgr2=0x7f00901006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090107460 0x7f00901006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 -- 172.21.15.38:0/3141582728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 msgr2=0x7f00901001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 0x7f00901001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008effd640 1 -- 172.21.15.38:0/3141582728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0090100f80 con 0x7f0090102200 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008e7fc640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 0x7f00901001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.404+0000 7f008f7fe640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090107460 0x7f00901006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.405+0000 7f008effd640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 0x7f0090101b70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f008000ba50 tx=0x7f008000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.405+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080019070 con 0x7f0090102200 2024-04-22T01:12:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.405+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008000cbe0 con 0x7f0090102200 2024-04-22T01:12:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.405+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00901a1c70 con 0x7f0090102200 2024-04-22T01:12:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.405+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00800076e0 con 0x7f0090102200 2024-04-22T01:12:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.405+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00901a2170 con 0x7f0090102200 2024-04-22T01:12:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.406+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00900fd6e0 con 0x7f0090102200 2024-04-22T01:12:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.407+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f008001c070 con 0x7f0090102200 2024-04-22T01:12:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.408+0000 7f0094806640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f00780a2b90 0x7f00780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.408+0000 7f008e7fc640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f00780a2b90 0x7f00780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.408+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f00800ceb60 con 0x7f0090102200 2024-04-22T01:12:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.408+0000 7f008e7fc640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f00780a2b90 0x7f00780a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0084006fd0 tx=0x7f0084008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:28.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.412+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f008009b6b0 con 0x7f0090102200 2024-04-22T01:12:28.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.597+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0090062370 con 0x7f0090102200 2024-04-22T01:12:28.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.598+0000 7f0094806640 1 -- 172.21.15.38:0/3141582728 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f008009afb0 con 0x7f0090102200 2024-04-22T01:12:28.598 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.599+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f00780a2b90 msgr2=0x7f00780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f00780a2b90 0x7f00780a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0084006fd0 tx=0x7f0084008040 comp rx=0 tx=0).stop 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 msgr2=0x7f0090101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 0x7f0090101b70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f008000ba50 tx=0x7f008000bf10 comp rx=0 tx=0).stop 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 shutdown_connections 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f00780a2b90 0x7f00780a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0090107460 0x7f00901006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0090104b30 0x7f00901001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 --2- 172.21.15.38:0/3141582728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0090102200 0x7f0090101b70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 >> 172.21.15.38:0/3141582728 conn(0x7f00900fbbf0 msgr2=0x7f0090102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 shutdown_connections 2024-04-22T01:12:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.600+0000 7f0095808640 1 -- 172.21.15.38:0/3141582728 wait complete. 2024-04-22T01:12:28.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:12:28.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 -- 172.21.15.38:0/1391159435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 msgr2=0x7fa39c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/1391159435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c073bd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa398009930 tx=0x7fa39802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 -- 172.21.15.38:0/1391159435 shutdown_connections 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/1391159435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c10dd60 0x7fa39c110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/1391159435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c0741a0 0x7fa39c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/1391159435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c073bd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 -- 172.21.15.38:0/1391159435 >> 172.21.15.38:0/1391159435 conn(0x7fa39c0fcd20 msgr2=0x7fa39c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 -- 172.21.15.38:0/1391159435 shutdown_connections 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.713+0000 7fa3a40e6640 1 -- 172.21.15.38:0/1391159435 wait complete. 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 Processor -- start 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 -- start start 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c0741a0 0x7fa39c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c10dd60 0x7fa39c1a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c113a00 con 0x7fa39c075780 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c113880 con 0x7fa39c0741a0 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a40e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c113b80 con 0x7fa39c10dd60 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49502/0 (socket says 172.21.15.38:49502) 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 -- 172.21.15.38:0/2903891952 learned_addr learned my addr 172.21.15.38:0/2903891952 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a2683640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c10dd60 0x7fa39c1a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1e82640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c0741a0 0x7fa39c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 -- 172.21.15.38:0/2903891952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c10dd60 msgr2=0x7fa39c1a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c10dd60 0x7fa39c1a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 -- 172.21.15.38:0/2903891952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c0741a0 msgr2=0x7fa39c19e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c0741a0 0x7fa39c19e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1681640 1 -- 172.21.15.38:0/2903891952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39c1a5cf0 con 0x7fa39c075780 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a2683640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c10dd60 0x7fa39c1a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.714+0000 7fa3a1e82640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c0741a0 0x7fa39c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.715+0000 7fa3a1681640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c19e610 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00d770 tx=0x7fa38c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.715+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c017070 con 0x7fa39c075780 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.715+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa38c005500 con 0x7fa39c075780 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.715+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa39c1a5fe0 con 0x7fa39c075780 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.715+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c004e00 con 0x7fa39c075780 2024-04-22T01:12:28.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.715+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa39c1a6450 con 0x7fa39c075780 2024-04-22T01:12:28.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.716+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa39c103f30 con 0x7fa39c075780 2024-04-22T01:12:28.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.717+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa38c004fa0 con 0x7fa39c075780 2024-04-22T01:12:28.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.718+0000 7fa392ffd640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3880a2b90 0x7fa3880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:28.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.718+0000 7fa3a1e82640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3880a2b90 0x7fa3880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:28.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.718+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa38c0cef60 con 0x7fa39c075780 2024-04-22T01:12:28.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.718+0000 7fa3a1e82640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3880a2b90 0x7fa3880a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3980096c0 tx=0x7fa398005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:28.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.722+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa38c09bab0 con 0x7fa39c075780 2024-04-22T01:12:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:28 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2649004615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:28 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2117794835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:28.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:28 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/901709548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:28.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:28 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2649004615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:28 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2117794835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:28.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:28 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/901709548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:28.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.946+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa39c062340 con 0x7fa39c075780 2024-04-22T01:12:28.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.947+0000 7fa392ffd640 1 -- 172.21.15.38:0/2903891952 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fa38c09b3b0 con 0x7fa39c075780 2024-04-22T01:12:28.947 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:28.947 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:11:00.429607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24451,24472,24446,24527,24485]},"id":1}]} 2024-04-22T01:12:28.948 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3880a2b90 msgr2=0x7fa3880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3880a2b90 0x7fa3880a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3980096c0 tx=0x7fa398005fd0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 msgr2=0x7fa39c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c19e610 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00d770 tx=0x7fa38c00dc30 comp rx=0 tx=0).stop 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 shutdown_connections 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3880a2b90 0x7fa3880a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa39c10dd60 0x7fa39c1a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa39c075780 0x7fa39c19e610 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 --2- 172.21.15.38:0/2903891952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c0741a0 0x7fa39c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 >> 172.21.15.38:0/2903891952 conn(0x7fa39c0fcd20 msgr2=0x7fa39c101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 shutdown_connections 2024-04-22T01:12:28.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:28.949+0000 7fa3a40e6640 1 -- 172.21.15.38:0/2903891952 wait complete. 2024-04-22T01:12:28.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:12:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:28 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2649004615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:28 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2117794835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:28 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/901709548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:29.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- 172.21.15.38:0/1391290139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc1006a0 msgr2=0x7f80bc100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- 172.21.15.38:0/1391290139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc1006a0 0x7f80bc100ac0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009900 tx=0x7f80b002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:29.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- 172.21.15.38:0/1391290139 shutdown_connections 2024-04-22T01:12:29.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- 172.21.15.38:0/1391290139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80bc102250 0x7f80bc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- 172.21.15.38:0/1391290139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80bc101890 0x7f80bc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- 172.21.15.38:0/1391290139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc1006a0 0x7f80bc100ac0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- 172.21.15.38:0/1391290139 >> 172.21.15.38:0/1391290139 conn(0x7f80bc0fbc30 msgr2=0x7f80bc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- 172.21.15.38:0/1391290139 shutdown_connections 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- 172.21.15.38:0/1391290139 wait complete. 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 Processor -- start 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- start start 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80bc1006a0 0x7f80bc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80bc101890 0x7f80bc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 0x7f80bc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc10c320 con 0x7f80bc102250 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc10c1a0 con 0x7f80bc101890 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.063+0000 7f80c3cae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc10c4a0 con 0x7f80bc1006a0 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 0x7f80bc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 0x7f80bc19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49522/0 (socket says 172.21.15.38:49522) 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 -- 172.21.15.38:0/4151909031 learned_addr learned my addr 172.21.15.38:0/4151909031 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c1a4a640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80bc1006a0 0x7f80bc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c1249640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80bc101890 0x7f80bc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 -- 172.21.15.38:0/4151909031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80bc1006a0 msgr2=0x7f80bc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80bc1006a0 0x7f80bc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 -- 172.21.15.38:0/4151909031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80bc101890 msgr2=0x7f80bc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80bc101890 0x7f80bc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 -- 172.21.15.38:0/4151909031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc19f6b0 con 0x7f80bc102250 2024-04-22T01:12:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c224b640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 0x7f80bc19f040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f80b400eaa0 tx=0x7f80b400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b400cdb0 con 0x7f80bc102250 2024-04-22T01:12:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b4005020 con 0x7f80bc102250 2024-04-22T01:12:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b40116b0 con 0x7f80bc102250 2024-04-22T01:12:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80bc19f9a0 con 0x7f80bc102250 2024-04-22T01:12:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.064+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80bc19ff40 con 0x7f80bc102250 2024-04-22T01:12:29.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.065+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80bc1a01d0 con 0x7f80bc102250 2024-04-22T01:12:29.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.067+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f80b4005560 con 0x7f80bc102250 2024-04-22T01:12:29.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.068+0000 7f80aeffd640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80a80a2c10 0x7f80a80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.068+0000 7f80c1a4a640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80a80a2c10 0x7f80a80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.068+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f80b40cf8a0 con 0x7f80bc102250 2024-04-22T01:12:29.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.068+0000 7f80c1a4a640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80a80a2c10 0x7f80a80a50d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80b0000c00 tx=0x7f80b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:29.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.072+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f80b4016030 con 0x7f80bc102250 2024-04-22T01:12:29.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.257+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80bc1a03a0 con 0x7f80bc102250 2024-04-22T01:12:29.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.259+0000 7f80aeffd640 1 -- 172.21.15.38:0/4151909031 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7f80b409be20 con 0x7f80bc102250 2024-04-22T01:12:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:29.260 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:12:29.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80a80a2c10 msgr2=0x7f80a80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80a80a2c10 0x7f80a80a50d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80b0000c00 tx=0x7f80b0005fd0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 msgr2=0x7f80bc19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 0x7f80bc19f040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f80b400eaa0 tx=0x7f80b400ef60 comp rx=0 tx=0).stop 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 shutdown_connections 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f80a80a2c10 0x7f80a80a50d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80bc102250 0x7f80bc19f040 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80bc101890 0x7f80bc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 --2- 172.21.15.38:0/4151909031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80bc1006a0 0x7f80bc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 >> 172.21.15.38:0/4151909031 conn(0x7f80bc0fbc30 msgr2=0x7f80bc0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 shutdown_connections 2024-04-22T01:12:29.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.261+0000 7f80c3cae640 1 -- 172.21.15.38:0/4151909031 wait complete. 2024-04-22T01:12:29.273 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:12:29.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.378+0000 7fc0f6440640 1 -- 172.21.15.38:0/697311647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0107460 msgr2=0x7fc0f01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.378+0000 7fc0f6440640 1 --2- 172.21.15.38:0/697311647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0107460 0x7fc0f01098b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009ca0 tx=0x7fc0e802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- 172.21.15.38:0/697311647 shutdown_connections 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 --2- 172.21.15.38:0/697311647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0107460 0x7fc0f01098b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 --2- 172.21.15.38:0/697311647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f0104b30 0x7fc0f0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 --2- 172.21.15.38:0/697311647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0102200 0x7fc0f01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- 172.21.15.38:0/697311647 >> 172.21.15.38:0/697311647 conn(0x7fc0f00fbbf0 msgr2=0x7fc0f00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- 172.21.15.38:0/697311647 shutdown_connections 2024-04-22T01:12:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- 172.21.15.38:0/697311647 wait complete. 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 Processor -- start 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- start start 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 0x7fc0f0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0104b30 0x7fc0f0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f0107460 0x7fc0f019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f010d420 con 0x7fc0f0102200 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f010d2a0 con 0x7fc0f0107460 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.379+0000 7fc0f6440640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f010d5a0 con 0x7fc0f0104b30 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 0x7fc0f0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 0x7fc0f0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49546/0 (socket says 172.21.15.38:49546) 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 -- 172.21.15.38:0/1983754487 learned_addr learned my addr 172.21.15.38:0/1983754487 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0f49dd640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f0107460 0x7fc0f019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0ef7fe640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0104b30 0x7fc0f0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 -- 172.21.15.38:0/1983754487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0104b30 msgr2=0x7fc0f0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0104b30 0x7fc0f0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 -- 172.21.15.38:0/1983754487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f0107460 msgr2=0x7fc0f019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f0107460 0x7fc0f019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 -- 172.21.15.38:0/1983754487 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f019f6b0 con 0x7fc0f0102200 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0ef7fe640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0104b30 0x7fc0f0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0effff640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 0x7fc0f0197a90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800d770 tx=0x7fc0d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8017070 con 0x7fc0f0102200 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0d8005500 con 0x7fc0f0102200 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8004e00 con 0x7fc0f0102200 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f019f9a0 con 0x7fc0f0102200 2024-04-22T01:12:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.380+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f019fef0 con 0x7fc0f0102200 2024-04-22T01:12:29.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.381+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f0068560 con 0x7fc0f0102200 2024-04-22T01:12:29.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.382+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc0d8004fa0 con 0x7fc0f0102200 2024-04-22T01:12:29.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.383+0000 7fc0ed7fa640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc0dc0a2c10 0x7fc0dc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.383+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fc0d80cf250 con 0x7fc0f0102200 2024-04-22T01:12:29.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.383+0000 7fc0ef7fe640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc0dc0a2c10 0x7fc0dc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.383+0000 7fc0ef7fe640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc0dc0a2c10 0x7fc0dc0a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0005d00 tx=0x7fc0e0009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:29.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.387+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc0d809be60 con 0x7fc0f0102200 2024-04-22T01:12:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.571+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0f002d170 con 0x7fc0f0102200 2024-04-22T01:12:29.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.572+0000 7fc0ed7fa640 1 -- 172.21.15.38:0/1983754487 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fc0f002d170 con 0x7fc0f0102200 2024-04-22T01:12:29.572 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:12:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc0dc0a2c10 msgr2=0x7fc0dc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc0dc0a2c10 0x7fc0dc0a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0005d00 tx=0x7fc0e0009550 comp rx=0 tx=0).stop 2024-04-22T01:12:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 msgr2=0x7fc0f0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 0x7fc0f0197a90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800d770 tx=0x7fc0d800dc30 comp rx=0 tx=0).stop 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 shutdown_connections 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc0dc0a2c10 0x7fc0dc0a50d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f0107460 0x7fc0f019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f0104b30 0x7fc0f0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 --2- 172.21.15.38:0/1983754487 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f0102200 0x7fc0f0197a90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 >> 172.21.15.38:0/1983754487 conn(0x7fc0f00fbbf0 msgr2=0x7fc0f0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 shutdown_connections 2024-04-22T01:12:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.574+0000 7fc0f6440640 1 -- 172.21.15.38:0/1983754487 wait complete. 2024-04-22T01:12:29.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 -- 172.21.15.38:0/3635374673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 msgr2=0x7fe2fc107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 --2- 172.21.15.38:0/3635374673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc107000 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009900 tx=0x7fe2f002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 -- 172.21.15.38:0/3635374673 shutdown_connections 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 --2- 172.21.15.38:0/3635374673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2fc107540 0x7fe2fc109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 --2- 172.21.15.38:0/3635374673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc107000 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 --2- 172.21.15.38:0/3635374673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 0x7fe2fc1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 -- 172.21.15.38:0/3635374673 >> 172.21.15.38:0/3635374673 conn(0x7fe2fc0fbc30 msgr2=0x7fe2fc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:29.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 -- 172.21.15.38:0/3635374673 shutdown_connections 2024-04-22T01:12:29.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.690+0000 7fe300fd6640 1 -- 172.21.15.38:0/3635374673 wait complete. 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 Processor -- start 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 -- start start 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 0x7fe2fc101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2fc107540 0x7fe2fc1007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc10d480 con 0x7fe2fc104c10 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc10d300 con 0x7fe2fc1022e0 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe300fd6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc10d600 con 0x7fe2fc107540 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe2f9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe2f9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc100290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49564/0 (socket says 172.21.15.38:49564) 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.691+0000 7fe2f9d9b640 1 -- 172.21.15.38:0/1627253011 learned_addr learned my addr 172.21.15.38:0/1627253011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2fa59c640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 0x7fe2fc101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2fad9d640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2fc107540 0x7fe2fc1007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2f9d9b640 1 -- 172.21.15.38:0/1627253011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2fc107540 msgr2=0x7fe2fc1007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2f9d9b640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2fc107540 0x7fe2fc1007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2f9d9b640 1 -- 172.21.15.38:0/1627253011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 msgr2=0x7fe2fc101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2f9d9b640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 0x7fe2fc101c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2f9d9b640 1 -- 172.21.15.38:0/1627253011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc101060 con 0x7fe2fc104c10 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2fa59c640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 0x7fe2fc101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2f9d9b640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc100290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0000c00 tx=0x7fe2f0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0043020 con 0x7fe2fc104c10 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2f00054e0 con 0x7fe2fc104c10 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2fc1a1c40 con 0x7fe2fc104c10 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0003e60 con 0x7fe2fc104c10 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.692+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2fc1a2190 con 0x7fe2fc104c10 2024-04-22T01:12:29.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.693+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2fc0fd720 con 0x7fe2fc104c10 2024-04-22T01:12:29.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.694+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe2f0046020 con 0x7fe2fc104c10 2024-04-22T01:12:29.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.695+0000 7fe2eb7fe640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe2e40a2c10 0x7fe2e40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:29.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.695+0000 7fe2fa59c640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe2e40a2c10 0x7fe2e40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:29.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.695+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe2f00f1680 con 0x7fe2fc104c10 2024-04-22T01:12:29.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.696+0000 7fe2fa59c640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe2e40a2c10 0x7fe2e40a50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005d00 tx=0x7fe2f4005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:29.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.700+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe2f004d020 con 0x7fe2fc104c10 2024-04-22T01:12:29.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.886+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe2fc062340 con 0x7fe2fc104c10 2024-04-22T01:12:29.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.886+0000 7fe2eb7fe640 1 -- 172.21.15.38:0/1627253011 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fe2f00be210 con 0x7fe2fc104c10 2024-04-22T01:12:29.887 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe2e40a2c10 msgr2=0x7fe2e40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe2e40a2c10 0x7fe2e40a50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005d00 tx=0x7fe2f4005c70 comp rx=0 tx=0).stop 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 msgr2=0x7fe2fc100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc100290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0000c00 tx=0x7fe2f0005dd0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 shutdown_connections 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe2e40a2c10 0x7fe2e40a50d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2fc107540 0x7fe2fc1007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2fc104c10 0x7fe2fc100290 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 --2- 172.21.15.38:0/1627253011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2fc1022e0 0x7fe2fc101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:29.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 >> 172.21.15.38:0/1627253011 conn(0x7fe2fc0fbc30 msgr2=0x7fe2fc102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:29.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 shutdown_connections 2024-04-22T01:12:29.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:29.888+0000 7fe300fd6640 1 -- 172.21.15.38:0/1627253011 wait complete. 2024-04-22T01:12:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:29 smithi111 ceph-mon[31057]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 197 B/s wr, 10 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:29 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3141582728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:29 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2903891952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:29 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4151909031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:29.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:29 smithi038 ceph-mon[28043]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 197 B/s wr, 10 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:29.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:29 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3141582728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:29.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:29 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2903891952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:29.899 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:29 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4151909031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:29.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:12:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 -- 172.21.15.38:0/1127850584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 msgr2=0x7f5360100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 --2- 172.21.15.38:0/1127850584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f5360100ac0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5350009900 tx=0x7f535002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 -- 172.21.15.38:0/1127850584 shutdown_connections 2024-04-22T01:12:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 --2- 172.21.15.38:0/1127850584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5360102250 0x7f5360108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 --2- 172.21.15.38:0/1127850584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5360101890 0x7f5360101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 --2- 172.21.15.38:0/1127850584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f5360100ac0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 -- 172.21.15.38:0/1127850584 >> 172.21.15.38:0/1127850584 conn(0x7f53600fbc30 msgr2=0x7f53600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 -- 172.21.15.38:0/1127850584 shutdown_connections 2024-04-22T01:12:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 -- 172.21.15.38:0/1127850584 wait complete. 2024-04-22T01:12:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 Processor -- start 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.005+0000 7f536613a640 1 -- start start 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f536010b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5360101890 0x7f5360109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5360102250 0x7f5360109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f536010d440 con 0x7f53601006a0 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f536010d2c0 con 0x7f5360101890 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f536010d5c0 con 0x7f5360102250 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f536010b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f536010b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49580/0 (socket says 172.21.15.38:49580) 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 -- 172.21.15.38:0/810770547 learned_addr learned my addr 172.21.15.38:0/810770547 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535effd640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5360101890 0x7f5360109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535ffff640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5360102250 0x7f5360109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 -- 172.21.15.38:0/810770547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5360102250 msgr2=0x7f5360109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5360102250 0x7f5360109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 -- 172.21.15.38:0/810770547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5360101890 msgr2=0x7f5360109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5360101890 0x7f5360109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 -- 172.21.15.38:0/810770547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536010a430 con 0x7f53601006a0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535f7fe640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f536010b020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5350005ab0 tx=0x7f5350005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350041070 con 0x7f53601006a0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53500054e0 con 0x7f53601006a0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53601a1b90 con 0x7f53601006a0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.006+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350034440 con 0x7f53601006a0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.007+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53601a2130 con 0x7f53601006a0 2024-04-22T01:12:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.007+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5360107a40 con 0x7f53601006a0 2024-04-22T01:12:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.009+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5350044070 con 0x7f53601006a0 2024-04-22T01:12:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.009+0000 7f535cff9640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53480a2c10 0x7f53480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.009+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f53500f1250 con 0x7f53601006a0 2024-04-22T01:12:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.009+0000 7f535effd640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53480a2c10 0x7f53480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.010+0000 7f535effd640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53480a2c10 0x7f53480a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5354005fd0 tx=0x7f5354005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:30.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.013+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f53500bdde0 con 0x7f53601006a0 2024-04-22T01:12:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:29 smithi203 ceph-mon[31825]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 197 B/s wr, 10 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-04-22T01:12:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:29 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3141582728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:29 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2903891952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:29 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4151909031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.200+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f536002d3e0 con 0x7f53601006a0 2024-04-22T01:12:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.201+0000 7f535cff9640 1 -- 172.21.15.38:0/810770547 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f53500bd6e0 con 0x7f53601006a0 2024-04-22T01:12:30.201 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.202+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53480a2c10 msgr2=0x7f53480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.202+0000 7f536613a640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53480a2c10 0x7f53480a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5354005fd0 tx=0x7f5354005850 comp rx=0 tx=0).stop 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.202+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 msgr2=0x7f536010b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.202+0000 7f536613a640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f536010b020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5350005ab0 tx=0x7f5350005dc0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.202+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 shutdown_connections 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53480a2c10 0x7f53480a50d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5360102250 0x7f5360109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5360101890 0x7f5360109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 --2- 172.21.15.38:0/810770547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53601006a0 0x7f536010b020 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 >> 172.21.15.38:0/810770547 conn(0x7f53600fbc30 msgr2=0x7f5360104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:30.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 shutdown_connections 2024-04-22T01:12:30.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.203+0000 7f536613a640 1 -- 172.21.15.38:0/810770547 wait complete. 2024-04-22T01:12:30.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:12:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 -- 172.21.15.38:0/2109781502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c102250 msgr2=0x7efe9c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 --2- 172.21.15.38:0/2109781502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c102250 0x7efe9c108ae0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efe94009ca0 tx=0x7efe9402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 -- 172.21.15.38:0/2109781502 shutdown_connections 2024-04-22T01:12:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 --2- 172.21.15.38:0/2109781502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c102250 0x7efe9c108ae0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 --2- 172.21.15.38:0/2109781502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe9c101890 0x7efe9c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 --2- 172.21.15.38:0/2109781502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe9c1006a0 0x7efe9c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 -- 172.21.15.38:0/2109781502 >> 172.21.15.38:0/2109781502 conn(0x7efe9c0fbc30 msgr2=0x7efe9c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 -- 172.21.15.38:0/2109781502 shutdown_connections 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 -- 172.21.15.38:0/2109781502 wait complete. 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 Processor -- start 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.319+0000 7efea466f640 1 -- start start 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 0x7efe9c06a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe9c101890 0x7efe9c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe9c102250 0x7efe9c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c10c380 con 0x7efe9c1006a0 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c10c200 con 0x7efe9c101890 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c10c500 con 0x7efe9c102250 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 0x7efe9c06a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 0x7efe9c06a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49596/0 (socket says 172.21.15.38:49596) 2024-04-22T01:12:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 -- 172.21.15.38:0/2723379911 learned_addr learned my addr 172.21.15.38:0/2723379911 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea1c0a640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe9c101890 0x7efe9c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 -- 172.21.15.38:0/2723379911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe9c102250 msgr2=0x7efe9c19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea2c0c640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe9c102250 0x7efe9c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe9c102250 0x7efe9c19ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 -- 172.21.15.38:0/2723379911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe9c101890 msgr2=0x7efe9c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe9c101890 0x7efe9c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 -- 172.21.15.38:0/2723379911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9c19f770 con 0x7efe9c1006a0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea240b640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 0x7efe9c06a8b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e9c0 tx=0x7efe9800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9800ccc0 con 0x7efe9c1006a0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe9c19fa00 con 0x7efe9c1006a0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe980035f0 con 0x7efe9c1006a0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9800ce60 con 0x7efe9c1006a0 2024-04-22T01:12:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.320+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe9c19ffa0 con 0x7efe9c1006a0 2024-04-22T01:12:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.321+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe9c1079c0 con 0x7efe9c1006a0 2024-04-22T01:12:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.322+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efe980037a0 con 0x7efe9c1006a0 2024-04-22T01:12:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.323+0000 7efe937fe640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe880a2b90 0x7efe880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.323+0000 7efea1c0a640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe880a2b90 0x7efe880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.323+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7efe980cf8f0 con 0x7efe9c1006a0 2024-04-22T01:12:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.324+0000 7efea1c0a640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe880a2b90 0x7efe880a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efe9c06a620 tx=0x7efe8c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.327+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efe9809c440 con 0x7efe9c1006a0 2024-04-22T01:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.516+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efe9c02d170 con 0x7efe9c1006a0 2024-04-22T01:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.516+0000 7efe937fe640 1 -- 172.21.15.38:0/2723379911 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7efe9809bd40 con 0x7efe9c1006a0 2024-04-22T01:12:30.518 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe880a2b90 msgr2=0x7efe880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe880a2b90 0x7efe880a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efe9c06a620 tx=0x7efe8c009110 comp rx=0 tx=0).stop 2024-04-22T01:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 msgr2=0x7efe9c06a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 0x7efe9c06a8b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e9c0 tx=0x7efe9800edc0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 shutdown_connections 2024-04-22T01:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe880a2b90 0x7efe880a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe9c102250 0x7efe9c19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe9c101890 0x7efe9c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 --2- 172.21.15.38:0/2723379911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe9c1006a0 0x7efe9c06a8b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 >> 172.21.15.38:0/2723379911 conn(0x7efe9c0fbc30 msgr2=0x7efe9c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:30.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 shutdown_connections 2024-04-22T01:12:30.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.518+0000 7efea466f640 1 -- 172.21.15.38:0/2723379911 wait complete. 2024-04-22T01:12:30.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 -- 172.21.15.38:0/3692845445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 msgr2=0x7f3ce4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 --2- 172.21.15.38:0/3692845445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce4101d10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009900 tx=0x7f3cd802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 -- 172.21.15.38:0/3692845445 shutdown_connections 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 --2- 172.21.15.38:0/3692845445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce4102250 0x7f3ce4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 --2- 172.21.15.38:0/3692845445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce4101d10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 --2- 172.21.15.38:0/3692845445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ce41006a0 0x7f3ce4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 -- 172.21.15.38:0/3692845445 >> 172.21.15.38:0/3692845445 conn(0x7f3ce40fbc30 msgr2=0x7f3ce40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 -- 172.21.15.38:0/3692845445 shutdown_connections 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.641+0000 7f3cec337640 1 -- 172.21.15.38:0/3692845445 wait complete. 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 Processor -- start 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 -- start start 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce41006a0 0x7f3ce4199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ce4102250 0x7f3ce41a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce410c320 con 0x7f3ce4101890 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce410c1a0 con 0x7f3ce4102250 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cec337640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce410c4a0 con 0x7f3ce41006a0 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3ce98d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3ce98d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49604/0 (socket says 172.21.15.38:49604) 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3ce98d2640 1 -- 172.21.15.38:0/4191042032 learned_addr learned my addr 172.21.15.38:0/4191042032 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cea0d3640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce41006a0 0x7f3ce4199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.642+0000 7f3cea8d4640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ce4102250 0x7f3ce41a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3ce98d2640 1 -- 172.21.15.38:0/4191042032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce41006a0 msgr2=0x7f3ce4199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3ce98d2640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce41006a0 0x7f3ce4199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3ce98d2640 1 -- 172.21.15.38:0/4191042032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ce4102250 msgr2=0x7f3ce41a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3ce98d2640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ce4102250 0x7f3ce41a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3ce98d2640 1 -- 172.21.15.38:0/4191042032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce41a18f0 con 0x7f3ce4101890 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3ce98d2640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce419a1f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009230 tx=0x7f3cd8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8041070 con 0x7f3ce4101890 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce41a1b80 con 0x7f3ce4101890 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cd80044d0 con 0x7f3ce4101890 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8034440 con 0x7f3ce4101890 2024-04-22T01:12:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.643+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce41a20c0 con 0x7f3ce4101890 2024-04-22T01:12:30.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.644+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce41a2350 con 0x7f3ce4101890 2024-04-22T01:12:30.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.645+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3cd8044070 con 0x7f3ce4101890 2024-04-22T01:12:30.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.646+0000 7f3cd77fe640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3cd00a2b90 0x7f3cd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:30.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.646+0000 7f3cea0d3640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3cd00a2b90 0x7f3cd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:30.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.646+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f3cd80f1460 con 0x7f3ce4101890 2024-04-22T01:12:30.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.646+0000 7f3cea0d3640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3cd00a2b90 0x7f3cd00a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3ce419ab80 tx=0x7f3ce000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:30.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.649+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3cd80bda80 con 0x7f3ce4101890 2024-04-22T01:12:30.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.872+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce41a2520 con 0x7f3ce4101890 2024-04-22T01:12:30.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.873+0000 7f3cd77fe640 1 -- 172.21.15.38:0/4191042032 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f3cd80c1d80 con 0x7f3ce4101890 2024-04-22T01:12:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:30.874 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:11:00.429607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24451,24472,24446,24527,24485]},"id":1}]} 2024-04-22T01:12:30.874 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:12:30.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3cd00a2b90 msgr2=0x7f3cd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3cd00a2b90 0x7f3cd00a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3ce419ab80 tx=0x7f3ce000b390 comp rx=0 tx=0).stop 2024-04-22T01:12:30.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 msgr2=0x7f3ce419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce419a1f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009230 tx=0x7f3cd8005d50 comp rx=0 tx=0).stop 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 shutdown_connections 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3cd00a2b90 0x7f3cd00a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ce4102250 0x7f3ce41a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4101890 0x7f3ce419a1f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 --2- 172.21.15.38:0/4191042032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce41006a0 0x7f3ce4199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 >> 172.21.15.38:0/4191042032 conn(0x7f3ce40fbc30 msgr2=0x7f3ce40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 shutdown_connections 2024-04-22T01:12:30.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.875+0000 7f3cec337640 1 -- 172.21.15.38:0/4191042032 wait complete. 2024-04-22T01:12:30.886 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:30 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1983754487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.887 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:30 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1627253011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.887 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:30 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/810770547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.887 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:30 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2723379911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:12:30.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:30 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1983754487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:30 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1627253011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:30 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/810770547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:30.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:30 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2723379911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:31.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 -- 172.21.15.38:0/3487902001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe714107540 msgr2=0x7fe714109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 --2- 172.21.15.38:0/3487902001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe714107540 0x7fe714109930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009ca0 tx=0x7fe70c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 -- 172.21.15.38:0/3487902001 shutdown_connections 2024-04-22T01:12:31.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 --2- 172.21.15.38:0/3487902001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe714107540 0x7fe714109930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 --2- 172.21.15.38:0/3487902001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714104c10 0x7fe714107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 --2- 172.21.15.38:0/3487902001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7141022e0 0x7fe7141046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 -- 172.21.15.38:0/3487902001 >> 172.21.15.38:0/3487902001 conn(0x7fe7140fbc30 msgr2=0x7fe7140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 -- 172.21.15.38:0/3487902001 shutdown_connections 2024-04-22T01:12:31.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 -- 172.21.15.38:0/3487902001 wait complete. 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 Processor -- start 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.998+0000 7fe71ad97640 1 -- start start 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 0x7fe714197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe714104c10 0x7fe714198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714107540 0x7fe71419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe71410d4a0 con 0x7fe7141022e0 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe71410d320 con 0x7fe714107540 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe71410d620 con 0x7fe714104c10 2024-04-22T01:12:31.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 0x7fe714197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 0x7fe714197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49622/0 (socket says 172.21.15.38:49622) 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 -- 172.21.15.38:0/344930878 learned_addr learned my addr 172.21.15.38:0/344930878 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe719334640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714107540 0x7fe71419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe713fff640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe714104c10 0x7fe714198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 -- 172.21.15.38:0/344930878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe714104c10 msgr2=0x7fe714198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe714104c10 0x7fe714198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 -- 172.21.15.38:0/344930878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714107540 msgr2=0x7fe71419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714107540 0x7fe71419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 -- 172.21.15.38:0/344930878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71419f700 con 0x7fe7141022e0 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe719334640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714107540 0x7fe71419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:31.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe713fff640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe714104c10 0x7fe714198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe718b33640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 0x7fe714197ae0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe70400b4a0 tx=0x7fe70400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704014070 con 0x7fe7141022e0 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe704010af0 con 0x7fe7141022e0 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71419f9f0 con 0x7fe7141022e0 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:30.999+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704017d10 con 0x7fe7141022e0 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.000+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe71419ff90 con 0x7fe7141022e0 2024-04-22T01:12:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.000+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7140fd720 con 0x7fe7141022e0 2024-04-22T01:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.001+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe704017070 con 0x7fe7141022e0 2024-04-22T01:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.002+0000 7fe711ffb640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6fc0a2c10 0x7fe6fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.002+0000 7fe713fff640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6fc0a2c10 0x7fe6fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.002+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe7040ce7f0 con 0x7fe7141022e0 2024-04-22T01:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.003+0000 7fe713fff640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6fc0a2c10 0x7fe6fc0a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe708005d00 tx=0x7fe708005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.006+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe70409b340 con 0x7fe7141022e0 2024-04-22T01:12:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:30 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1983754487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:30 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1627253011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:31.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:30 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/810770547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:31.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:30 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2723379911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:31.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.192+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe714062340 con 0x7fe7141022e0 2024-04-22T01:12:31.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.193+0000 7fe711ffb640 1 -- 172.21.15.38:0/344930878 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7fe70409ac40 con 0x7fe7141022e0 2024-04-22T01:12:31.194 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:31.194 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:12:31.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6fc0a2c10 msgr2=0x7fe6fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6fc0a2c10 0x7fe6fc0a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe708005d00 tx=0x7fe708005ed0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 msgr2=0x7fe714197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 0x7fe714197ae0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe70400b4a0 tx=0x7fe70400b960 comp rx=0 tx=0).stop 2024-04-22T01:12:31.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 shutdown_connections 2024-04-22T01:12:31.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6fc0a2c10 0x7fe6fc0a50d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe714107540 0x7fe71419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe714104c10 0x7fe714198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 --2- 172.21.15.38:0/344930878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7141022e0 0x7fe714197ae0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.195+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 >> 172.21.15.38:0/344930878 conn(0x7fe7140fbc30 msgr2=0x7fe714102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.196+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 shutdown_connections 2024-04-22T01:12:31.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.196+0000 7fe71ad97640 1 -- 172.21.15.38:0/344930878 wait complete. 2024-04-22T01:12:31.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 -- 172.21.15.38:0/3749759688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264101890 msgr2=0x7ff264101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 --2- 172.21.15.38:0/3749759688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264101890 0x7ff264101d10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff250009900 tx=0x7ff25002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 -- 172.21.15.38:0/3749759688 shutdown_connections 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 --2- 172.21.15.38:0/3749759688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff264102250 0x7ff264108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 --2- 172.21.15.38:0/3749759688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264101890 0x7ff264101d10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 --2- 172.21.15.38:0/3749759688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2641006a0 0x7ff264100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 -- 172.21.15.38:0/3749759688 >> 172.21.15.38:0/3749759688 conn(0x7ff2640fbc30 msgr2=0x7ff2640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 -- 172.21.15.38:0/3749759688 shutdown_connections 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 -- 172.21.15.38:0/3749759688 wait complete. 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.310+0000 7ff26b137640 1 Processor -- start 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 -- start start 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2641006a0 0x7ff26406a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff264101890 0x7ff26419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 0x7ff26419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff26410c350 con 0x7ff264102250 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff26410c1d0 con 0x7ff264101890 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff26b137640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff26410c4d0 con 0x7ff2641006a0 2024-04-22T01:12:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 0x7ff26419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 0x7ff26419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49634/0 (socket says 172.21.15.38:49634) 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 -- 172.21.15.38:0/1490303467 learned_addr learned my addr 172.21.15.38:0/1490303467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff268ed3640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2641006a0 0x7ff26406a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff263fff640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff264101890 0x7ff26419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 -- 172.21.15.38:0/1490303467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2641006a0 msgr2=0x7ff26406a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2641006a0 0x7ff26406a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 -- 172.21.15.38:0/1490303467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff264101890 msgr2=0x7ff26419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff264101890 0x7ff26419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 -- 172.21.15.38:0/1490303467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26419f6e0 con 0x7ff264102250 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.311+0000 7ff2696d4640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 0x7ff26419ee20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cad0 tx=0x7ff25c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.312+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00ae20 con 0x7ff264102250 2024-04-22T01:12:31.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.312+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c005020 con 0x7ff264102250 2024-04-22T01:12:31.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.312+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26419f9d0 con 0x7ff264102250 2024-04-22T01:12:31.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.312+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00f6f0 con 0x7ff264102250 2024-04-22T01:12:31.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.312+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26419fea0 con 0x7ff264102250 2024-04-22T01:12:31.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.313+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff264107960 con 0x7ff264102250 2024-04-22T01:12:31.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.314+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff25c0051d0 con 0x7ff264102250 2024-04-22T01:12:31.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.314+0000 7ff261ffb640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff2540a2b90 0x7ff2540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.314+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff25c0cf380 con 0x7ff264102250 2024-04-22T01:12:31.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.314+0000 7ff268ed3640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff2540a2b90 0x7ff2540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.315+0000 7ff268ed3640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff2540a2b90 0x7ff2540a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff2641016f0 tx=0x7ff258005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.319+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff25c014030 con 0x7ff264102250 2024-04-22T01:12:31.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.545+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff26402d170 con 0x7ff264102250 2024-04-22T01:12:31.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.546+0000 7ff261ffb640 1 -- 172.21.15.38:0/1490303467 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7ff25c09b960 con 0x7ff264102250 2024-04-22T01:12:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:31.546 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:11:00.429607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24451,24472,24446,24527,24485]},"id":1}]} 2024-04-22T01:12:31.547 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:12:31.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff2540a2b90 msgr2=0x7ff2540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff2540a2b90 0x7ff2540a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff2641016f0 tx=0x7ff258005f90 comp rx=0 tx=0).stop 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 msgr2=0x7ff26419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 0x7ff26419ee20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cad0 tx=0x7ff25c00cf90 comp rx=0 tx=0).stop 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 shutdown_connections 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff2540a2b90 0x7ff2540a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff264102250 0x7ff26419ee20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff264101890 0x7ff26419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 --2- 172.21.15.38:0/1490303467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2641006a0 0x7ff26406a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 >> 172.21.15.38:0/1490303467 conn(0x7ff2640fbc30 msgr2=0x7ff264104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 shutdown_connections 2024-04-22T01:12:31.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.548+0000 7ff26b137640 1 -- 172.21.15.38:0/1490303467 wait complete. 2024-04-22T01:12:31.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T01:10:29.943110+0000', 'modified': '2024-04-22T01:11:00.429607+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24527, 'mds_1': 24485, 'mds_2': 24472, 'mds_3': 24451, 'mds_4': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14595': {'gid': 14595, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6839/1149927748', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 1149927748}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 1149927748}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14607': {'gid': 14607, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6841/3631879100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6840', 'nonce': 3631879100}, {'type': 'v1', 'addr': '172.21.15.38:6841', 'nonce': 3631879100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/3645664141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 3645664141}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 3645664141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6833/1598672021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6832', 'nonce': 1598672021}, {'type': 'v1', 'addr': '172.21.15.111:6833', 'nonce': 1598672021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/4027190739', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 4027190739}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 4027190739}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.111:6835/2761116669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 2761116669}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 2761116669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6837/2993816469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 2993816469}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 2993816469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.203:6839/765839756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 765839756}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 765839756}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/1636882248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 1636882248}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 1636882248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3001874105', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3001874105}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3001874105}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24451, 24472, 24446, 24527, 24485]} 2024-04-22T01:12:31.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T01:12:31.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 -- 172.21.15.38:0/2150707561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf41006a0 msgr2=0x7ffaf4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 --2- 172.21.15.38:0/2150707561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf41006a0 0x7ffaf4100ac0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009900 tx=0x7ffae802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 -- 172.21.15.38:0/2150707561 shutdown_connections 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 --2- 172.21.15.38:0/2150707561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 0x7ffaf4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 --2- 172.21.15.38:0/2150707561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf4101890 0x7ffaf4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 --2- 172.21.15.38:0/2150707561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf41006a0 0x7ffaf4100ac0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 -- 172.21.15.38:0/2150707561 >> 172.21.15.38:0/2150707561 conn(0x7ffaf40fbc30 msgr2=0x7ffaf40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 -- 172.21.15.38:0/2150707561 shutdown_connections 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.663+0000 7ffafaba0640 1 -- 172.21.15.38:0/2150707561 wait complete. 2024-04-22T01:12:31.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 Processor -- start 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 -- start start 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf41006a0 0x7ffaf4104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 0x7ffaf4102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 0x7ffaf41031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf410c440 con 0x7ffaf4101890 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf410c2c0 con 0x7ffaf4102250 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffafaba0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf410c5c0 con 0x7ffaf41006a0 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 0x7ffaf4102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 0x7ffaf4102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49644/0 (socket says 172.21.15.38:49644) 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf3fff640 1 -- 172.21.15.38:0/92486000 learned_addr learned my addr 172.21.15.38:0/92486000 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf893c640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf41006a0 0x7ffaf4104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf913d640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 0x7ffaf41031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf3fff640 1 -- 172.21.15.38:0/92486000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf41006a0 msgr2=0x7ffaf4104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.664+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf41006a0 0x7ffaf4104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf3fff640 1 -- 172.21.15.38:0/92486000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 msgr2=0x7ffaf41031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 0x7ffaf41031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf3fff640 1 -- 172.21.15.38:0/92486000 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf4103a40 con 0x7ffaf4101890 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf913d640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 0x7ffaf41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 0x7ffaf4102c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffae000b4a0 tx=0x7ffae000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf1ffb640 1 -- 172.21.15.38:0/92486000 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0014070 con 0x7ffaf4101890 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf1ffb640 1 -- 172.21.15.38:0/92486000 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffae0010af0 con 0x7ffaf4101890 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf41a1cc0 con 0x7ffaf4101890 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffaf1ffb640 1 -- 172.21.15.38:0/92486000 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0017d10 con 0x7ffaf4101890 2024-04-22T01:12:31.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.665+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf41a21c0 con 0x7ffaf4101890 2024-04-22T01:12:31.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.667+0000 7ffaf1ffb640 1 -- 172.21.15.38:0/92486000 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ffae0017070 con 0x7ffaf4101890 2024-04-22T01:12:31.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.667+0000 7ffafd782740 1 Processor -- start 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 -- start start 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf1ffb640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ffae40a2b90 0x7ffae40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13070 0x55656be13490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be18560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be18bd0 0x55656be1b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55656bd77df0 con 0x55656be13a60 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55656bd20640 con 0x55656be13070 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffafd782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55656bd73db0 con 0x55656be18bd0 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf913d640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ffae40a2b90 0x7ffae40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be18560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be18560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49656/0 (socket says 172.21.15.38:49656) 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 -- 172.21.15.38:0/2496291542 learned_addr learned my addr 172.21.15.38:0/2496291542 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf1ffb640 1 -- 172.21.15.38:0/92486000 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ffae00ce960 con 0x7ffaf4101890 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf893c640 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13070 0x55656be13490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 -- 172.21.15.38:0/2496291542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be18bd0 msgr2=0x55656be1b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be18bd0 0x55656be1b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 -- 172.21.15.38:0/2496291542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13070 msgr2=0x55656be13490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13070 0x55656be13490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 -- 172.21.15.38:0/2496291542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55656ba8c2d0 con 0x55656be13a60 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf913d640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ffae40a2b90 0x7ffae40a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffaec007fa0 tx=0x7ffaec007900 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.668+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be18560 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffae009ad40 tx=0x7ffae00504c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626db21b15dafa03 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffadeffd640 1 -- 172.21.15.38:0/2496291542 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae001e070 con 0x55656be13a60 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffadeffd640 1 -- 172.21.15.38:0/2496291542 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffae009b4b0 con 0x55656be13a60 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffadeffd640 1 -- 172.21.15.38:0/2496291542 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0017360 con 0x55656be13a60 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 -- 172.21.15.38:0/2496291542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 msgr2=0x55656be18560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be18560 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffae009ad40 tx=0x7ffae00504c0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 -- 172.21.15.38:0/2496291542 shutdown_connections 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be18bd0 0x55656be1b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be18560 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 --2- 172.21.15.38:0/2496291542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13070 0x55656be13490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 -- 172.21.15.38:0/2496291542 >> 172.21.15.38:0/2496291542 conn(0x55656be0e5e0 msgr2=0x55656be0e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 -- 172.21.15.38:0/2496291542 shutdown_connections 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.669+0000 7ffafd782740 1 -- 172.21.15.38:0/2496291542 wait complete. 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 -- start start 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 Processor -- start 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be1d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be1db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be24b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55656be1bb80 con 0x55656be13a60 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55656bd33be0 con 0x55656be18bd0 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55656be1bd00 con 0x55656be13070 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffafd782740 10 client.0 _ll_register_callbacks cb 0x55656bd335d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be1db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be1db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49660/0 (socket says 172.21.15.38:49660) 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1884058988 learned_addr learned my addr 172.21.15.38:0/1884058988 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf893c640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be1d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1884058988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 msgr2=0x55656be1d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be1d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 msgr2=0x55656be24b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf913d640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be24b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be24b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1884058988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55656bb566b0 con 0x55656be13a60 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.670+0000 7ffaf893c640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be1d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be1db20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffae0004810 tx=0x7ffae0099140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e8a62fca4f730 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffad3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffad3fff640 1 -- 172.21.15.38:0/1884058988 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae001e030 con 0x55656be13a60 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffad3fff640 1 -- 172.21.15.38:0/1884058988 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ffae0010ca0 con 0x55656be13a60 2024-04-22T01:12:31.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffae009b4b0 con 0x55656be13a60 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffad3fff640 1 -- 172.21.15.38:0/1884058988 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffae00b8050 con 0x55656be13a60 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 10 client.15201 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55656be27b40 con 0x55656be13a60 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffad3fff640 1 -- 172.21.15.38:0/1884058988 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffae009b4b0 con 0x55656be13a60 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffad3fff640 1 -- 172.21.15.38:0/1884058988 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ffae0096b00 con 0x55656be13a60 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 10 client.15201 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 10 client.15201 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 20 client.15201 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be29ca0 0x55656be2b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 4 client.15201 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.671+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55656bd85330 con 0x55656be29ca0 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.672+0000 7ffaf893c640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be29ca0 0x55656be2b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.672+0000 7ffad3fff640 10 client.15201 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:31.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.672+0000 7ffaf893c640 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be29ca0 0x55656be2b110 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=26491a6979a64991 server_cookie=5113719962c1bca2 in_seq=0 out_seq=0 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffad3fff640 1 -- 172.21.15.38:0/1884058988 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55656bd85330 con 0x55656be29ca0 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffad3fff640 10 client.15201 handle_command_reply: tid=0 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 client.15201 shutdown 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 msgr2=0x55656be1db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be1db20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffae0004810 tx=0x7ffae0099140 comp rx=0 tx=0).stop 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 shutdown_connections 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be29ca0 0x55656be2b110 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be24b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be1db20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffad3fff640 0 client.15201 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 --2- 172.21.15.38:0/1884058988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be1d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 >> 172.21.15.38:0/1884058988 conn(0x55656be0e5e0 msgr2=0x55656be0d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 shutdown_connections 2024-04-22T01:12:31.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 1 -- 172.21.15.38:0/1884058988 wait complete. 2024-04-22T01:12:31.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.673+0000 7ffafd782740 20 client.15201 trim_cache size 0 max 16384 2024-04-22T01:12:31.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.691+0000 7ffafd782740 1 Processor -- start 2024-04-22T01:12:31.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.691+0000 7ffafd782740 1 -- start start 2024-04-22T01:12:31.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffafd782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be14410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffafd782740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13a60 0x55656be24b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffafd782740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 0x55656be0e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffafd782740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55656be29450 con 0x55656be18bd0 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffafd782740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55656be29b20 con 0x55656be13a60 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffafd782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55656be2a1f0 con 0x55656be13070 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 0x55656be0e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 0x55656be0e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49670/0 (socket says 172.21.15.38:49670) 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 -- 172.21.15.38:0/2423556222 learned_addr learned my addr 172.21.15.38:0/2423556222 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13a60 0x55656be24b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf893c640 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be14410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 -- 172.21.15.38:0/2423556222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 msgr2=0x55656be14410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be14410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 -- 172.21.15.38:0/2423556222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13a60 msgr2=0x55656be24b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13a60 0x55656be24b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 -- 172.21.15.38:0/2423556222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55656be13e40 con 0x55656be18bd0 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf893c640 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be14410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffaf913d640 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 0x55656be0e9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0117b0 tx=0x7ffaec011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c497225a2a47d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffade7fc640 1 -- 172.21.15.38:0/2423556222 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec019040 con 0x55656be18bd0 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffade7fc640 1 -- 172.21.15.38:0/2423556222 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaec023070 con 0x55656be18bd0 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.692+0000 7ffade7fc640 1 -- 172.21.15.38:0/2423556222 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec01ec00 con 0x55656be18bd0 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 -- 172.21.15.38:0/2423556222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 msgr2=0x55656be0e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 0x55656be0e9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0117b0 tx=0x7ffaec011000 comp rx=0 tx=0).stop 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 -- 172.21.15.38:0/2423556222 shutdown_connections 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be18bd0 0x55656be0e9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be13a60 0x55656be24b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 --2- 172.21.15.38:0/2423556222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be14410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 -- 172.21.15.38:0/2423556222 >> 172.21.15.38:0/2423556222 conn(0x55656be0e5e0 msgr2=0x55656be09340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 -- 172.21.15.38:0/2423556222 shutdown_connections 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 -- 172.21.15.38:0/2423556222 wait complete. 2024-04-22T01:12:31.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 -- start start 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.693+0000 7ffafd782740 1 Processor -- start 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be24310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be24850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be20db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55656be2af90 con 0x55656be13a60 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55656be2a8c0 con 0x55656be18bd0 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55656be2b660 con 0x55656be13070 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 10 client.0 _ll_register_callbacks cb 0x7ffae009aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be24850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be24850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49672/0 (socket says 172.21.15.38:49672) 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1487884531 learned_addr learned my addr 172.21.15.38:0/1487884531 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf913d640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be20db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1487884531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 msgr2=0x55656be24310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be24310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 msgr2=0x55656be20db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be20db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 -- 172.21.15.38:0/1487884531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55656be21420 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf913d640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be20db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffaf3fff640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be24850 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffae0097780 tx=0x7ffae00a3a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8952644adb63011 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffad3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffad3fff640 1 -- 172.21.15.38:0/1487884531 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae001e030 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffad3fff640 1 -- 172.21.15.38:0/1487884531 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ffae0096b00 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec023070 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffad3fff640 1 -- 172.21.15.38:0/1487884531 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffae009a8f0 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 10 client.15207 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.694+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55656be90760 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffad3fff640 1 -- 172.21.15.38:0/1487884531 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaec023070 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffad3fff640 1 -- 172.21.15.38:0/1487884531 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ffae0099e00 con 0x55656be13a60 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffafd782740 10 client.15207 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:31.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffafd782740 10 client.15207 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffafd782740 20 client.15207 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffafd782740 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be95170 0x55656be93e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffafd782740 4 client.15207 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "status"}) -- 0x55656bd7f680 con 0x55656be95170 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.695+0000 7ffaf893c640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be95170 0x55656be93e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffad3fff640 10 client.15207 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffaf893c640 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be95170 0x55656be93e80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=80c6364927155bbc server_cookie=ca0b2b791e7cdbb6 in_seq=0 out_seq=0 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffad3fff640 1 -- 172.21.15.38:0/1487884531 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55656bd7f680 con 0x55656be95170 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffad3fff640 10 client.15207 handle_command_reply: tid=0 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 client.15207 shutdown 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 msgr2=0x55656be24850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be24850 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffae0097780 tx=0x7ffae00a3a20 comp rx=0 tx=0).stop 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 shutdown_connections 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55656be95170 0x55656be93e80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55656be18bd0 0x55656be20db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffad3fff640 0 client.15207 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55656be13a60 0x55656be24850 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 --2- 172.21.15.38:0/1487884531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656be13070 0x55656be24310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 >> 172.21.15.38:0/1487884531 conn(0x55656be0e5e0 msgr2=0x55656be14410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 shutdown_connections 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 1 -- 172.21.15.38:0/1487884531 wait complete. 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.696+0000 7ffafd782740 20 client.15207 trim_cache size 0 max 16384 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 4, 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24446, 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 91.779337170000005, 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 145.54958400800001 2024-04-22T01:12:31.704 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ffae40a2b90 msgr2=0x7ffae40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ffae40a2b90 0x7ffae40a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffaec007fa0 tx=0x7ffaec007900 comp rx=0 tx=0).stop 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 msgr2=0x7ffaf4102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 0x7ffaf4102c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffae000b4a0 tx=0x7ffae000b960 comp rx=0 tx=0).stop 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 shutdown_connections 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ffae40a2b90 0x7ffae40a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffaf4102250 0x7ffaf41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf4101890 0x7ffaf4102c70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 --2- 172.21.15.38:0/92486000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffaf41006a0 0x7ffaf4104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 >> 172.21.15.38:0/92486000 conn(0x7ffaf40fbc30 msgr2=0x7ffaf40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 shutdown_connections 2024-04-22T01:12:31.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.697+0000 7ffafaba0640 1 -- 172.21.15.38:0/92486000 wait complete. 2024-04-22T01:12:31.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 -- 172.21.15.38:0/2892032727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48100680 msgr2=0x7f7d48100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 --2- 172.21.15.38:0/2892032727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48100680 0x7f7d48100aa0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009900 tx=0x7f7d3802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 -- 172.21.15.38:0/2892032727 shutdown_connections 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 --2- 172.21.15.38:0/2892032727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d48102230 0x7f7d48108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 --2- 172.21.15.38:0/2892032727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d48101870 0x7f7d48101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 --2- 172.21.15.38:0/2892032727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48100680 0x7f7d48100aa0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 -- 172.21.15.38:0/2892032727 >> 172.21.15.38:0/2892032727 conn(0x7f7d480fbbf0 msgr2=0x7f7d480fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 -- 172.21.15.38:0/2892032727 shutdown_connections 2024-04-22T01:12:31.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.811+0000 7f7d4d239640 1 -- 172.21.15.38:0/2892032727 wait complete. 2024-04-22T01:12:31.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.812+0000 7f7d4d239640 1 Processor -- start 2024-04-22T01:12:31.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.812+0000 7f7d4d239640 1 -- start start 2024-04-22T01:12:31.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.812+0000 7f7d4d239640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d48100680 0x7f7d48197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.812+0000 7f7d4d239640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 0x7f7d48198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4d239640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d48102230 0x7f7d4819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4d239640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810c2d0 con 0x7f7d48101870 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4d239640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810c150 con 0x7f7d48100680 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4d239640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810c450 con 0x7f7d48102230 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 0x7f7d48198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 0x7f7d48198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49686/0 (socket says 172.21.15.38:49686) 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 -- 172.21.15.38:0/1870609587 learned_addr learned my addr 172.21.15.38:0/1870609587 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d48100680 0x7f7d48197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 -- 172.21.15.38:0/1870609587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d48102230 msgr2=0x7f7d4819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d48102230 0x7f7d4819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 -- 172.21.15.38:0/1870609587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d48100680 msgr2=0x7f7d48197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d48100680 0x7f7d48197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 -- 172.21.15.38:0/1870609587 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4819f770 con 0x7f7d48101870 2024-04-22T01:12:31.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4659c640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 0x7f7d48198090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000d950 tx=0x7f7d3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d3ffff640 1 -- 172.21.15.38:0/1870609587 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d30017070 con 0x7f7d48101870 2024-04-22T01:12:31.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d3ffff640 1 -- 172.21.15.38:0/1870609587 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d30009c30 con 0x7f7d48101870 2024-04-22T01:12:31.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4819fa60 con 0x7f7d48101870 2024-04-22T01:12:31.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d3ffff640 1 -- 172.21.15.38:0/1870609587 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000f6b0 con 0x7f7d48101870 2024-04-22T01:12:31.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.813+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4819ffb0 con 0x7f7d48101870 2024-04-22T01:12:31.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.815+0000 7f7d3ffff640 1 -- 172.21.15.38:0/1870609587 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7d300109d0 con 0x7f7d48101870 2024-04-22T01:12:31.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 Processor -- start 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d3ffff640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d340a2b90 0x7f7d340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 -- start start 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4759e640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d340a2b90 0x7f7d340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d3ffff640 1 -- 172.21.15.38:0/1870609587 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7d300cf150 con 0x7f7d48101870 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e698a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e6e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e71470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561691dc9df0 con 0x561691e69480 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561691d72640 con 0x561691e69e70 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.816+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561691dc5db0 con 0x561691e6efe0 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e698a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49698/0 (socket says 172.21.15.38:49698) 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 -- 172.21.15.38:0/4236188038 learned_addr learned my addr 172.21.15.38:0/4236188038 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d4659c640 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e6e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d4759e640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d340a2b90 0x7f7d340a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7d40005fd0 tx=0x7f7d40009440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 -- 172.21.15.38:0/4236188038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 msgr2=0x561691e71470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e71470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 -- 172.21.15.38:0/4236188038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 msgr2=0x561691e6e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e6e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 -- 172.21.15.38:0/4236188038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561691ade2d0 con 0x561691e69480 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e698a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7d38000c00 tx=0x7f7d380095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ad1d148dee9e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d3d7fa640 1 -- 172.21.15.38:0/4236188038 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3802cb80 con 0x561691e69480 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d3d7fa640 1 -- 172.21.15.38:0/4236188038 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d38005040 con 0x561691e69480 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.817+0000 7f7d3d7fa640 1 -- 172.21.15.38:0/4236188038 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d38004920 con 0x561691e69480 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 -- 172.21.15.38:0/4236188038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 msgr2=0x561691e698a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e698a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7d38000c00 tx=0x7f7d380095e0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 -- 172.21.15.38:0/4236188038 shutdown_connections 2024-04-22T01:12:31.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e71470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e6e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/4236188038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e698a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 -- 172.21.15.38:0/4236188038 >> 172.21.15.38:0/4236188038 conn(0x561691e605e0 msgr2=0x561691e609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 -- 172.21.15.38:0/4236188038 shutdown_connections 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 -- 172.21.15.38:0/4236188038 wait complete. 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.818+0000 7f7d4fe25740 1 -- start start 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 Processor -- start 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e738e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e73e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e7ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561691e71f30 con 0x561691e6efe0 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561691d85be0 con 0x561691e69e70 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561691e720b0 con 0x561691e69480 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4fe25740 10 client.0 _ll_register_callbacks cb 0x561691d855d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e7ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e7ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49706/0 (socket says 172.21.15.38:49706) 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 -- 172.21.15.38:0/519799503 learned_addr learned my addr 172.21.15.38:0/519799503 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4659c640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e73e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 -- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 msgr2=0x561691e738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 -- 172.21.15.38:0/519799503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 msgr2=0x561691e73e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e73e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 -- 172.21.15.38:0/519799503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561691ba86b0 con 0x561691e6efe0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4659c640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e73e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d4759e640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e7ae60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009fd0 tx=0x7f7d40010ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b138239cf476655b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.819+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/519799503 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40021030 con 0x561691e6efe0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/519799503 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7d40004b20 con 0x561691e6efe0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d38005040 con 0x561691e6efe0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/519799503 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d4002b050 con 0x561691e6efe0 2024-04-22T01:12:31.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 10 client.15219 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561691e7df00 con 0x561691e6efe0 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/519799503 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d38005040 con 0x561691e6efe0 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/519799503 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7d4001d580 con 0x561691e6efe0 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 10 client.15219 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 10 client.15219 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 20 client.15219 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691e80620 0x561691e81a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 4 client.15219 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561691dd7330 con 0x561691e80620 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.820+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691e80620 0x561691e81a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.821+0000 7f7d2e7fc640 10 client.15219 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.821+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691e80620 0x561691e81a90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2c801182e5fbc9d0 server_cookie=7e809365dc99977d in_seq=0 out_seq=0 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/519799503 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561691dd7330 con 0x561691e80620 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d2e7fc640 10 client.15219 handle_command_reply: tid=0 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 client.15219 shutdown 2024-04-22T01:12:31.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 msgr2=0x561691e7ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e7ae60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009fd0 tx=0x7f7d40010ab0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 shutdown_connections 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691e80620 0x561691e81a90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e7ae60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e73e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d2e7fc640 0 client.15219 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/519799503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e738e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 >> 172.21.15.38:0/519799503 conn(0x561691e605e0 msgr2=0x561691e5f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 shutdown_connections 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 1 -- 172.21.15.38:0/519799503 wait complete. 2024-04-22T01:12:31.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.822+0000 7f7d4fe25740 20 client.15219 trim_cache size 0 max 16384 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.840+0000 7f7d4fe25740 1 Processor -- start 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.840+0000 7f7d4fe25740 1 -- start start 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e7adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e5ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561691e7fdd0 con 0x561691e6efe0 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561691e804a0 con 0x561691e69e70 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561691e80b70 con 0x561691e69480 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e5ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e5ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49720/0 (socket says 172.21.15.38:49720) 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 -- 172.21.15.38:0/3070735183 learned_addr learned my addr 172.21.15.38:0/3070735183 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 -- 172.21.15.38:0/3070735183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 msgr2=0x561691e609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 -- 172.21.15.38:0/3070735183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 msgr2=0x561691e7adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e7adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 -- 172.21.15.38:0/3070735183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561691e81dc0 con 0x561691e6efe0 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d4759e640 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e5ec20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000db40 tx=0x7f7d40004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc47d6ecafc33c6c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d3cff9640 1 -- 172.21.15.38:0/3070735183 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40033070 con 0x561691e6efe0 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d3cff9640 1 -- 172.21.15.38:0/3070735183 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d40016030 con 0x561691e6efe0 2024-04-22T01:12:31.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.841+0000 7f7d3cff9640 1 -- 172.21.15.38:0/3070735183 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40021040 con 0x561691e6efe0 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3070735183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 msgr2=0x561691e5ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e5ec20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000db40 tx=0x7f7d40004d60 comp rx=0 tx=0).stop 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3070735183 shutdown_connections 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e6efe0 0x561691e5ec20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e7adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3070735183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e69480 0x561691e609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3070735183 >> 172.21.15.38:0/3070735183 conn(0x561691e605e0 msgr2=0x561691e5b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3070735183 shutdown_connections 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3070735183 wait complete. 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.842+0000 7f7d4fe25740 1 -- start start 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 Processor -- start 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e76b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e77080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e7a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561691e81910 con 0x561691e69480 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561691e81240 con 0x561691e69e70 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561691e81fe0 con 0x561691e6efe0 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 10 client.0 _ll_register_callbacks cb 0x7f7d40009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e76b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e76b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49734/0 (socket says 172.21.15.38:49734) 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 -- 172.21.15.38:0/3849809802 learned_addr learned my addr 172.21.15.38:0/3849809802 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4759e640 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e7a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 -- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 msgr2=0x561691e7a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e7a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 -- 172.21.15.38:0/3849809802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 msgr2=0x561691e77080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e77080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 -- 172.21.15.38:0/3849809802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561691e7a690 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d46d9d640 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e76b40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7d38005ec0 tx=0x7f7d3802c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ce7cfdb7a9e9b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/3849809802 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3804edc0 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/3849809802 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7d38004030 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.843+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d40016030 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/3849809802 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d38004230 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 10 client.15225 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561691ee6c30 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/3849809802 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d40016030 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/3849809802 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7d3804e440 con 0x561691e69480 2024-04-22T01:12:31.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 10 client.15225 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 10 client.15225 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 20 client.15225 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691eeb390 0x561691ee77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 4 client.15225 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "status"}) -- 0x561691dd1680 con 0x561691eeb390 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.844+0000 7f7d4659c640 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691eeb390 0x561691ee77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d2e7fc640 10 client.15225 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4659c640 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691eeb390 0x561691ee77d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=926c1399214756f6 server_cookie=d509ec91a8f1069 in_seq=0 out_seq=0 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d2e7fc640 1 -- 172.21.15.38:0/3849809802 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561691dd1680 con 0x561691eeb390 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d2e7fc640 10 client.15225 handle_command_reply: tid=0 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 client.15225 shutdown 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 msgr2=0x561691e76b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e76b40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7d38005ec0 tx=0x7f7d3802c2b0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 shutdown_connections 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x561691eeb390 0x561691ee77d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561691e6efe0 0x561691e7a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d2e7fc640 0 client.15225 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561691e69e70 0x561691e77080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 --2- 172.21.15.38:0/3849809802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561691e69480 0x561691e76b40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 >> 172.21.15.38:0/3849809802 conn(0x561691e605e0 msgr2=0x561691e609c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 shutdown_connections 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 1 -- 172.21.15.38:0/3849809802 wait complete. 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.845+0000 7f7d4fe25740 20 client.15225 trim_cache size 0 max 16384 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 3, 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24451, 2024-04-22T01:12:31.851 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 92.942122482000002, 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 140.07464216100001 2024-04-22T01:12:31.852 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d340a2b90 msgr2=0x7f7d340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d340a2b90 0x7f7d340a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7d40005fd0 tx=0x7f7d40009440 comp rx=0 tx=0).stop 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 msgr2=0x7f7d48198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 0x7f7d48198090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000d950 tx=0x7f7d3000de10 comp rx=0 tx=0).stop 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 shutdown_connections 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7d340a2b90 0x7f7d340a5050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d48102230 0x7f7d4819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d48101870 0x7f7d48198090 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 --2- 172.21.15.38:0/1870609587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d48100680 0x7f7d48197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 >> 172.21.15.38:0/1870609587 conn(0x7f7d480fbbf0 msgr2=0x7f7d48068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 shutdown_connections 2024-04-22T01:12:31.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.846+0000 7f7d4d239640 1 -- 172.21.15.38:0/1870609587 wait complete. 2024-04-22T01:12:31.856 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:31 smithi038 ceph-mon[28043]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 170 B/s wr, 8 op/s; 55 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T01:12:31.856 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4191042032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:31.856 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/344930878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:31.856 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:31 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1490303467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:31.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T01:12:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:31 smithi111 ceph-mon[31057]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 170 B/s wr, 8 op/s; 55 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T01:12:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:31 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4191042032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:31 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/344930878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:31 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1490303467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- 172.21.15.38:0/869497823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 msgr2=0x7f4ca8106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- 172.21.15.38:0/869497823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8106f20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009900 tx=0x7f4c9402bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- 172.21.15.38:0/869497823 shutdown_connections 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- 172.21.15.38:0/869497823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8107460 0x7f4ca81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- 172.21.15.38:0/869497823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8106f20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- 172.21.15.38:0/869497823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8102200 0x7f4ca81045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- 172.21.15.38:0/869497823 >> 172.21.15.38:0/869497823 conn(0x7f4ca80fbbf0 msgr2=0x7f4ca80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- 172.21.15.38:0/869497823 shutdown_connections 2024-04-22T01:12:31.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- 172.21.15.38:0/869497823 wait complete. 2024-04-22T01:12:31.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 Processor -- start 2024-04-22T01:12:31.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- start start 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102200 0x7f4ca8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8107460 0x7f4ca819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca810d410 con 0x7f4ca8104b30 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca810d290 con 0x7f4ca8102200 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.960+0000 7f4caedfe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca810d590 con 0x7f4ca8107460 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49766/0 (socket says 172.21.15.38:49766) 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 -- 172.21.15.38:0/2543875480 learned_addr learned my addr 172.21.15.38:0/2543875480 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102200 0x7f4ca8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4cad39b640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8107460 0x7f4ca819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 -- 172.21.15.38:0/2543875480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8107460 msgr2=0x7f4ca819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8107460 0x7f4ca819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 -- 172.21.15.38:0/2543875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102200 msgr2=0x7f4ca8197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102200 0x7f4ca8197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 -- 172.21.15.38:0/2543875480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca819f760 con 0x7f4ca8104b30 2024-04-22T01:12:31.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8198080 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4c94000c00 tx=0x7f4c9402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca5ffb640 1 -- 172.21.15.38:0/2543875480 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94041070 con 0x7f4ca8104b30 2024-04-22T01:12:31.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca5ffb640 1 -- 172.21.15.38:0/2543875480 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c94004450 con 0x7f4ca8104b30 2024-04-22T01:12:31.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca819f9f0 con 0x7f4ca8104b30 2024-04-22T01:12:31.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4ca5ffb640 1 -- 172.21.15.38:0/2543875480 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94034440 con 0x7f4ca8104b30 2024-04-22T01:12:31.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.961+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca819ff30 con 0x7f4ca8104b30 2024-04-22T01:12:31.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.963+0000 7f4ca5ffb640 1 -- 172.21.15.38:0/2543875480 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4c94044070 con 0x7f4ca8104b30 2024-04-22T01:12:31.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 Processor -- start 2024-04-22T01:12:31.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 -- start start 2024-04-22T01:12:31.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b1e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b21470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563641a79df0 con 0x563641b19480 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563641a22640 con 0x563641b19e70 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4ca5ffb640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c980a2c10 0x7f4c980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563641a75db0 con 0x563641b1efe0 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cad39b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cad39b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b198a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49782/0 (socket says 172.21.15.38:49782) 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cad39b640 1 -- 172.21.15.38:0/1846064892 learned_addr learned my addr 172.21.15.38:0/1846064892 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cad39b640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c980a2c10 0x7f4c980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4ca5ffb640 1 -- 172.21.15.38:0/2543875480 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4c940f0ee0 con 0x7f4ca8104b30 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b1e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.964+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b21470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 -- 172.21.15.38:0/1846064892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 msgr2=0x563641b21470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b21470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 -- 172.21.15.38:0/1846064892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 msgr2=0x563641b1e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b1e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 -- 172.21.15.38:0/1846064892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56364178e2d0 con 0x563641b19480 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b21470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b198a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0014e70 tx=0x7f4ca00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3825ac4f8285e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4cad39b640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c980a2c10 0x7f4c980a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009fd0 tx=0x7f4ca001c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4c92ffd640 1 -- 172.21.15.38:0/1846064892 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0022070 con 0x563641b19480 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4c92ffd640 1 -- 172.21.15.38:0/1846064892 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca000ccb0 con 0x563641b19480 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.965+0000 7f4c92ffd640 1 -- 172.21.15.38:0/1846064892 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0005280 con 0x563641b19480 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- 172.21.15.38:0/1846064892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 msgr2=0x563641b198a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b198a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0014e70 tx=0x7f4ca00117b0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- 172.21.15.38:0/1846064892 shutdown_connections 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b21470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b1e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/1846064892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b198a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- 172.21.15.38:0/1846064892 >> 172.21.15.38:0/1846064892 conn(0x563641b105e0 msgr2=0x563641b109e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- 172.21.15.38:0/1846064892 shutdown_connections 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- 172.21.15.38:0/1846064892 wait complete. 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- start start 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 Processor -- start 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b239f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b23f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b2af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563641b21f30 con 0x563641b1efe0 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563641a35be0 con 0x563641b19e70 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.966+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563641b220b0 con 0x563641b19480 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cb19e1740 10 client.0 _ll_register_callbacks cb 0x563641a355d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b2af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b2af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49794/0 (socket says 172.21.15.38:49794) 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 -- 172.21.15.38:0/4201650546 learned_addr learned my addr 172.21.15.38:0/4201650546 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b23f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b239f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 -- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 msgr2=0x563641b239f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b239f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 -- 172.21.15.38:0/4201650546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 msgr2=0x563641b23f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b23f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 -- 172.21.15.38:0/4201650546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636418586b0 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cad39b640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b2af70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00052f0 tx=0x7f4ca0005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60961a72baede7ea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4c87fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4c87fff640 1 -- 172.21.15.38:0/4201650546 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0022070 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4c87fff640 1 -- 172.21.15.38:0/4201650546 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4ca0005750 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ca000ccb0 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4c87fff640 1 -- 172.21.15.38:0/4201650546 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ca0037050 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cb19e1740 10 client.15237 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.967+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563641b2df30 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4c87fff640 1 -- 172.21.15.38:0/4201650546 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca000ccb0 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4c87fff640 1 -- 172.21.15.38:0/4201650546 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4ca001d070 con 0x563641b1efe0 2024-04-22T01:12:31.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cb19e1740 10 client.15237 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cb19e1740 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cb19e1740 20 client.15237 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b30270 0x563641b316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cb19e1740 4 client.15237 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563641a87330 con 0x563641b30270 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.968+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b30270 0x563641b316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.969+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b30270 0x563641b316e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b48938dfe7c7d720 server_cookie=dc7b051aaf2fbd8 in_seq=0 out_seq=0 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.969+0000 7f4c87fff640 10 client.15237 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.969+0000 7f4c87fff640 1 -- 172.21.15.38:0/4201650546 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563641a87330 con 0x563641b30270 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.969+0000 7f4c87fff640 10 client.15237 handle_command_reply: tid=0 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 client.15237 shutdown 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 msgr2=0x563641b2af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b2af70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00052f0 tx=0x7f4ca0005320 comp rx=0 tx=0).stop 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 shutdown_connections 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b30270 0x563641b316e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b2af70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b23f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/4201650546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b239f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 >> 172.21.15.38:0/4201650546 conn(0x563641b105e0 msgr2=0x563641b0f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4c87fff640 0 client.15237 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:31.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 shutdown_connections 2024-04-22T01:12:31.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 1 -- 172.21.15.38:0/4201650546 wait complete. 2024-04-22T01:12:31.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.970+0000 7f4cb19e1740 20 client.15237 trim_cache size 0 max 16384 2024-04-22T01:12:31.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 Processor -- start 2024-04-22T01:12:31.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 -- start start 2024-04-22T01:12:31.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b0ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b0f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b389b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563641b2fa20 con 0x563641b1efe0 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563641b300f0 con 0x563641b19e70 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.988+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563641b307c0 con 0x563641b19480 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b389b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49810/0 (socket says 172.21.15.38:49810) 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 -- 172.21.15.38:0/3160198550 learned_addr learned my addr 172.21.15.38:0/3160198550 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b0ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 -- 172.21.15.38:0/3160198550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 msgr2=0x563641b0ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b0f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b0ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 -- 172.21.15.38:0/3160198550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 msgr2=0x563641b0f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b0f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 -- 172.21.15.38:0/3160198550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563641b0ec20 con 0x563641b1efe0 2024-04-22T01:12:31.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b0f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4cad39b640 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b389b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00117f0 tx=0x7f4ca0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b04f4af18daa00 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4c927fc640 1 -- 172.21.15.38:0/3160198550 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0022070 con 0x563641b1efe0 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4c927fc640 1 -- 172.21.15.38:0/3160198550 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca0005390 con 0x563641b1efe0 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.989+0000 7f4c927fc640 1 -- 172.21.15.38:0/3160198550 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca000fa50 con 0x563641b1efe0 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 -- 172.21.15.38:0/3160198550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 msgr2=0x563641b389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b389b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00117f0 tx=0x7f4ca0009580 comp rx=0 tx=0).stop 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 -- 172.21.15.38:0/3160198550 shutdown_connections 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b1efe0 0x563641b389b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b0f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/3160198550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b19480 0x563641b0ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 -- 172.21.15.38:0/3160198550 >> 172.21.15.38:0/3160198550 conn(0x563641b105e0 msgr2=0x563641b1a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 -- 172.21.15.38:0/3160198550 shutdown_connections 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 -- 172.21.15.38:0/3160198550 wait complete. 2024-04-22T01:12:31.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 -- start start 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.990+0000 7f4cb19e1740 1 Processor -- start 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b282d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b28810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b29760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563641b31560 con 0x563641b19480 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563641b30e90 con 0x563641b19e70 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563641b31c30 con 0x563641b1efe0 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 10 client.0 _ll_register_callbacks cb 0x7f4ca0004d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b282d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49818/0 (socket says 172.21.15.38:49818) 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 -- 172.21.15.38:0/609433467 learned_addr learned my addr 172.21.15.38:0/609433467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b28810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 -- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 msgr2=0x563641b29760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b29760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 -- 172.21.15.38:0/609433467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 msgr2=0x563641b28810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b28810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 -- 172.21.15.38:0/609433467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563641b29d50 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b28810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cacb9a640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b282d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x563641b1a270 tx=0x7f4c9c009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5286499212785573 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4c87fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4c87fff640 1 -- 172.21.15.38:0/609433467 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c024060 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ca0005390 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4c87fff640 1 -- 172.21.15.38:0/609433467 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4c9c004460 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.991+0000 7f4c87fff640 1 -- 172.21.15.38:0/609433467 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c9c02b050 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 10 client.15243 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563641b968a0 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4c87fff640 1 -- 172.21.15.38:0/609433467 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca0005390 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4c87fff640 1 -- 172.21.15.38:0/609433467 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4c9c01d3b0 con 0x563641b19480 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 10 client.15243 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 10 client.15243 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 20 client.15243 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b9b550 0x563641b9a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 4 client.15243 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "status"}) -- 0x563641a81680 con 0x563641b9b550 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.992+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b9b550 0x563641b9a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4c87fff640 10 client.15243 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4ca7fff640 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b9b550 0x563641b9a260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2dda87b6949e5096 server_cookie=8d7b977e2441a117 in_seq=0 out_seq=0 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4c87fff640 1 -- 172.21.15.38:0/609433467 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563641a81680 con 0x563641b9b550 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4c87fff640 10 client.15243 handle_command_reply: tid=0 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 client.15243 shutdown 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 msgr2=0x563641b282d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b282d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x563641b1a270 tx=0x7f4c9c009d60 comp rx=0 tx=0).stop 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 shutdown_connections 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563641b9b550 0x563641b9a260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563641b1efe0 0x563641b29760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4c87fff640 0 client.15243 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563641b19e70 0x563641b28810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 --2- 172.21.15.38:0/609433467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563641b19480 0x563641b282d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 >> 172.21.15.38:0/609433467 conn(0x563641b105e0 msgr2=0x563641b0ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 shutdown_connections 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 1 -- 172.21.15.38:0/609433467 wait complete. 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.993+0000 7f4cb19e1740 20 client.15243 trim_cache size 0 max 16384 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24472, 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:32.000 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 94.098413883000006, 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 134.217835175 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:32.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c980a2c10 msgr2=0x7f4c980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c980a2c10 0x7f4c980a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009fd0 tx=0x7f4ca001c000 comp rx=0 tx=0).stop 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 msgr2=0x7f4ca8198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8198080 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4c94000c00 tx=0x7f4c9402c640 comp rx=0 tx=0).stop 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 shutdown_connections 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4c980a2c10 0x7f4c980a50d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca8107460 0x7f4ca819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ca8104b30 0x7f4ca8198080 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 --2- 172.21.15.38:0/2543875480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca8102200 0x7f4ca8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 >> 172.21.15.38:0/2543875480 conn(0x7f4ca80fbbf0 msgr2=0x7f4ca8102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 shutdown_connections 2024-04-22T01:12:32.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:31.994+0000 7f4caedfe640 1 -- 172.21.15.38:0/2543875480 wait complete. 2024-04-22T01:12:32.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T01:12:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:31 smithi203 ceph-mon[31825]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 170 B/s wr, 8 op/s; 55 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T01:12:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:31 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4191042032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:31 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/344930878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:32.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:31 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1490303467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 -- 172.21.15.38:0/2413622639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 msgr2=0x7f09ec102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 --2- 172.21.15.38:0/2413622639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec102cf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009900 tx=0x7f09d402bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 -- 172.21.15.38:0/2413622639 shutdown_connections 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 --2- 172.21.15.38:0/2413622639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec103230 0x7f09ec109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 --2- 172.21.15.38:0/2413622639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec102cf0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 --2- 172.21.15.38:0/2413622639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec101680 0x7f09ec101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 -- 172.21.15.38:0/2413622639 >> 172.21.15.38:0/2413622639 conn(0x7f09ec0fcc10 msgr2=0x7f09ec0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 -- 172.21.15.38:0/2413622639 shutdown_connections 2024-04-22T01:12:32.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.107+0000 7f09f17a2640 1 -- 172.21.15.38:0/2413622639 wait complete. 2024-04-22T01:12:32.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 Processor -- start 2024-04-22T01:12:32.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 -- start start 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101680 0x7f09ec197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec103230 0x7f09ec19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09ec10d2e0 con 0x7f09ec102870 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f09ec10d160 con 0x7f09ec101680 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.108+0000 7f09f17a2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09ec10d460 con 0x7f09ec103230 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49844/0 (socket says 172.21.15.38:49844) 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3289956555 learned_addr learned my addr 172.21.15.38:0/3289956555 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101680 0x7f09ec197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec103230 0x7f09ec19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3289956555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec103230 msgr2=0x7f09ec19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec103230 0x7f09ec19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3289956555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101680 msgr2=0x7f09ec197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101680 0x7f09ec197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3289956555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ec19f5a0 con 0x7f09ec102870 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec197ec0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009230 tx=0x7f09d4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09e97fa640 1 -- 172.21.15.38:0/3289956555 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4041070 con 0x7f09ec102870 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09e97fa640 1 -- 172.21.15.38:0/3289956555 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d40054e0 con 0x7f09ec102870 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ec19f830 con 0x7f09ec102870 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09e97fa640 1 -- 172.21.15.38:0/3289956555 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4034440 con 0x7f09ec102870 2024-04-22T01:12:32.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.109+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ec19fd20 con 0x7f09ec102870 2024-04-22T01:12:32.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09e97fa640 1 -- 172.21.15.38:0/3289956555 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f09d4044070 con 0x7f09ec102870 2024-04-22T01:12:32.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 Processor -- start 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 -- start start 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec68faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec694b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec697670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5600ec5efdf0 con 0x5600ec690070 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5600ec598640 con 0x5600ec68f680 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f55df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5600ec5ebdb0 con 0x5600ec6951e0 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09ebfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec694b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09ebfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec694b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49860/0 (socket says 172.21.15.38:49860) 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09ebfff640 1 -- 172.21.15.38:0/3928694883 learned_addr learned my addr 172.21.15.38:0/3928694883 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec68faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09e97fa640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f09d80a2b90 0x7f09d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.112+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec697670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09ebfff640 1 -- 172.21.15.38:0/3928694883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 msgr2=0x5600ec697670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec697670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09ebfff640 1 -- 172.21.15.38:0/3928694883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 msgr2=0x5600ec68faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec68faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09ebfff640 1 -- 172.21.15.38:0/3928694883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600ec3042d0 con 0x5600ec690070 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec68faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f09d80a2b90 0x7f09d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09e97fa640 1 -- 172.21.15.38:0/3289956555 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f09d40f10d0 con 0x7f09ec102870 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec694b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00bcb0 tx=0x7f09dc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd767fa8679fe97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09e27fc640 1 -- 172.21.15.38:0/3928694883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc004020 con 0x5600ec690070 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09e27fc640 1 -- 172.21.15.38:0/3928694883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc005500 con 0x5600ec690070 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09e27fc640 1 -- 172.21.15.38:0/3928694883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc004d80 con 0x5600ec690070 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.113+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f09d80a2b90 0x7f09d80a5050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5600ec6908d0 tx=0x7f09e4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 -- 172.21.15.38:0/3928694883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 msgr2=0x5600ec694b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec694b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00bcb0 tx=0x7f09dc0107a0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 -- 172.21.15.38:0/3928694883 shutdown_connections 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec697670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec694b70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 --2- 172.21.15.38:0/3928694883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec68faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 -- 172.21.15.38:0/3928694883 >> 172.21.15.38:0/3928694883 conn(0x5600ec6867e0 msgr2=0x5600ec686be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 -- 172.21.15.38:0/3928694883 shutdown_connections 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 -- 172.21.15.38:0/3928694883 wait complete. 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 -- start start 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.114+0000 7f09f55df740 1 Processor -- start 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 0x5600ec699b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec690070 0x5600ec69a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec6a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5600ec698190 con 0x5600ec68f680 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5600ec5abbe0 con 0x5600ec690070 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5600ec698310 con 0x5600ec6951e0 2024-04-22T01:12:32.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 10 client.0 _ll_register_callbacks cb 0x5600ec5ab5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 0x5600ec699b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 0x5600ec699b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49874/0 (socket says 172.21.15.38:49874) 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 -- 172.21.15.38:0/2272098973 learned_addr learned my addr 172.21.15.38:0/2272098973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec690070 0x5600ec69a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec6a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 -- 172.21.15.38:0/2272098973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 msgr2=0x5600ec6a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec6a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 -- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec690070 msgr2=0x5600ec69a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec690070 0x5600ec69a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 -- 172.21.15.38:0/2272098973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600ec3ce6b0 con 0x5600ec68f680 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec6a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09ebfff640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 0x5600ec699b60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f09dc01a000 tx=0x7f09dc00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfa50eb2dc758b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09cb7fe640 1 -- 172.21.15.38:0/2272098973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc02a060 con 0x5600ec68f680 2024-04-22T01:12:32.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09cb7fe640 1 -- 172.21.15.38:0/2272098973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f09dc004310 con 0x5600ec68f680 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.115+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09dc005500 con 0x5600ec68f680 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09cb7fe640 1 -- 172.21.15.38:0/2272098973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09dc031050 con 0x5600ec68f680 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 10 client.15255 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5600ec6a40c0 con 0x5600ec68f680 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09cb7fe640 1 -- 172.21.15.38:0/2272098973 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc005500 con 0x5600ec68f680 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09cb7fe640 1 -- 172.21.15.38:0/2272098973 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f09dc004d80 con 0x5600ec68f680 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 10 client.15255 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 10 client.15255 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 20 client.15255 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec6a6360 0x5600ec6a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 4 client.15255 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600ec5fd330 con 0x5600ec6a6360 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.116+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec6a6360 0x5600ec6a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.117+0000 7f09cb7fe640 10 client.15255 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.117+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec6a6360 0x5600ec6a77d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c71b2db4f642320b server_cookie=99053edaeab0a4cf in_seq=0 out_seq=0 2024-04-22T01:12:32.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09cb7fe640 1 -- 172.21.15.38:0/2272098973 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600ec5fd330 con 0x5600ec6a6360 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09cb7fe640 10 client.15255 handle_command_reply: tid=0 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 client.15255 shutdown 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 msgr2=0x5600ec699b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 0x5600ec699b60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f09dc01a000 tx=0x7f09dc00bfd0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 shutdown_connections 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec6a6360 0x5600ec6a77d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec6a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec690070 0x5600ec69a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09cb7fe640 0 client.15255 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 --2- 172.21.15.38:0/2272098973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec68f680 0x5600ec699b60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 >> 172.21.15.38:0/2272098973 conn(0x5600ec6867e0 msgr2=0x5600ec6859e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 shutdown_connections 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 1 -- 172.21.15.38:0/2272098973 wait complete. 2024-04-22T01:12:32.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.118+0000 7f09f55df740 20 client.15255 trim_cache size 0 max 16384 2024-04-22T01:12:32.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 Processor -- start 2024-04-22T01:12:32.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 -- start start 2024-04-22T01:12:32.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec686bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec690070 0x5600ec6a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 0x5600ec685150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5600ec6a5b10 con 0x5600ec6951e0 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5600ec6a61e0 con 0x5600ec68f680 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f55df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5600ec6a68b0 con 0x5600ec690070 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f0fa1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 0x5600ec685150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f0fa1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 0x5600ec685150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49884/0 (socket says 172.21.15.38:49884) 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.136+0000 7f09f0fa1640 1 -- 172.21.15.38:0/565715583 learned_addr learned my addr 172.21.15.38:0/565715583 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09ebfff640 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec686bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec690070 0x5600ec6a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09f0fa1640 1 -- 172.21.15.38:0/565715583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec690070 msgr2=0x5600ec6a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec690070 0x5600ec6a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09f0fa1640 1 -- 172.21.15.38:0/565715583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 msgr2=0x5600ec686bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec686bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09f0fa1640 1 -- 172.21.15.38:0/565715583 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600ec6947e0 con 0x5600ec6951e0 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec690070 0x5600ec6a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:32.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 0x5600ec685150 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f09e401e5a0 tx=0x7f09e4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7af7a362446a4fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09e1ffb640 1 -- 172.21.15.38:0/565715583 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4012920 con 0x5600ec6951e0 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09e1ffb640 1 -- 172.21.15.38:0/565715583 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e4027480 con 0x5600ec6951e0 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.137+0000 7f09e1ffb640 1 -- 172.21.15.38:0/565715583 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e401b680 con 0x5600ec6951e0 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- 172.21.15.38:0/565715583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 msgr2=0x5600ec685150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 0x5600ec685150 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f09e401e5a0 tx=0x7f09e4013000 comp rx=0 tx=0).stop 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- 172.21.15.38:0/565715583 shutdown_connections 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec6951e0 0x5600ec685150 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec690070 0x5600ec6a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- 172.21.15.38:0/565715583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec686bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- 172.21.15.38:0/565715583 >> 172.21.15.38:0/565715583 conn(0x5600ec6867e0 msgr2=0x5600ec6986e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- 172.21.15.38:0/565715583 shutdown_connections 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- 172.21.15.38:0/565715583 wait complete. 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- start start 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 Processor -- start 2024-04-22T01:12:32.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec69c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec69cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec69fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5600ec6a7650 con 0x5600ec690070 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5600ec6a6f80 con 0x5600ec68f680 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.138+0000 7f09f55df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5600ec6a7d20 con 0x5600ec6951e0 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09f55df740 10 client.0 _ll_register_callbacks cb 0x7f09dc0056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec69cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec69cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49888/0 (socket says 172.21.15.38:49888) 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3977688222 learned_addr learned my addr 172.21.15.38:0/3977688222 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec69c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09f0fa1640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec69fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3977688222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 msgr2=0x5600ec69fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec69fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 msgr2=0x5600ec69c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec69c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 -- 172.21.15.38:0/3977688222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600ec6a03e0 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09eb7fe640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec69cdd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f09d40ee800 tx=0x7f09d40a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8820b8e418147515 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09cb7fe640 1 -- 172.21.15.38:0/3977688222 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d404b040 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09cb7fe640 1 -- 172.21.15.38:0/3977688222 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f09d40ee910 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09e4027480 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09cb7fe640 1 -- 172.21.15.38:0/3977688222 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09d4004c60 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09f55df740 10 client.15261 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5600ec70c3d0 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.139+0000 7f09cb7fe640 1 -- 172.21.15.38:0/3977688222 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09e4027480 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09cb7fe640 1 -- 172.21.15.38:0/3977688222 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f09d40b9bd0 con 0x5600ec690070 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09f55df740 10 client.15261 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09f55df740 10 client.15261 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09f55df740 20 client.15261 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09f55df740 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec710f80 0x5600ec70fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09f55df740 4 client.15261 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "status"}) -- 0x5600ec5f7680 con 0x5600ec710f80 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec710f80 0x5600ec70fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09cb7fe640 10 client.15261 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.140+0000 7f09ebfff640 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec710f80 0x5600ec70fc90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5cd5d1c240268f8c server_cookie=105ccbe43728ee57 in_seq=0 out_seq=0 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09cb7fe640 1 -- 172.21.15.38:0/3977688222 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5600ec5f7680 con 0x5600ec710f80 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09cb7fe640 10 client.15261 handle_command_reply: tid=0 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 client.15261 shutdown 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 msgr2=0x5600ec69cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec69cdd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f09d40ee800 tx=0x7f09d40a8040 comp rx=0 tx=0).stop 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 shutdown_connections 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x5600ec710f80 0x5600ec70fc90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600ec6951e0 0x5600ec69fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600ec690070 0x5600ec69cdd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09cb7fe640 0 client.15261 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 --2- 172.21.15.38:0/3977688222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600ec68f680 0x5600ec69c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 >> 172.21.15.38:0/3977688222 conn(0x5600ec6867e0 msgr2=0x5600ec686bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 shutdown_connections 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 1 -- 172.21.15.38:0/3977688222 wait complete. 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f55df740 20 client.15261 trim_cache size 0 max 16384 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24485, 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 95.261752133000002, 2024-04-22T01:12:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 137.35449323700001 2024-04-22T01:12:32.149 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f09d80a2b90 msgr2=0x7f09d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f17a2640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f09d80a2b90 0x7f09d80a5050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5600ec6908d0 tx=0x7f09e4004a40 comp rx=0 tx=0).stop 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 msgr2=0x7f09ec197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.141+0000 7f09f17a2640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec197ec0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009230 tx=0x7f09d4005dd0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 shutdown_connections 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f09d80a2b90 0x7f09d80a5050 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec103230 0x7f09ec19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09ec102870 0x7f09ec197ec0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 --2- 172.21.15.38:0/3289956555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101680 0x7f09ec197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 >> 172.21.15.38:0/3289956555 conn(0x7f09ec0fcc10 msgr2=0x7f09ec105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 shutdown_connections 2024-04-22T01:12:32.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.142+0000 7f09f17a2640 1 -- 172.21.15.38:0/3289956555 wait complete. 2024-04-22T01:12:32.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 -- 172.21.15.38:0/1630979756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 msgr2=0x7f3864109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 --2- 172.21.15.38:0/1630979756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f3864109be0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3854009900 tx=0x7f385402bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 -- 172.21.15.38:0/1630979756 shutdown_connections 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 --2- 172.21.15.38:0/1630979756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f3864109be0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 --2- 172.21.15.38:0/1630979756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 0x7f3864102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 --2- 172.21.15.38:0/1630979756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38641017a0 0x7f3864101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 -- 172.21.15.38:0/1630979756 >> 172.21.15.38:0/1630979756 conn(0x7f38640fcd10 msgr2=0x7f38640ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 -- 172.21.15.38:0/1630979756 shutdown_connections 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.255+0000 7f386ac78640 1 -- 172.21.15.38:0/1630979756 wait complete. 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 Processor -- start 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 -- start start 2024-04-22T01:12:32.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38641017a0 0x7f3864197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 0x7f3864197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f386419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f386410d400 con 0x7f3864103350 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f386410d280 con 0x7f38641017a0 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f386ac78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f386410d580 con 0x7f3864102990 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f3869215640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f386419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f3869215640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f386419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49912/0 (socket says 172.21.15.38:49912) 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f3869215640 1 -- 172.21.15.38:0/3156593641 learned_addr learned my addr 172.21.15.38:0/3156593641 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.256+0000 7f3863fff640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 0x7f3864197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3868a14640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38641017a0 0x7f3864197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3869215640 1 -- 172.21.15.38:0/3156593641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 msgr2=0x7f3864197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3869215640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 0x7f3864197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3869215640 1 -- 172.21.15.38:0/3156593641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38641017a0 msgr2=0x7f3864197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3869215640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38641017a0 0x7f3864197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3869215640 1 -- 172.21.15.38:0/3156593641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386419f640 con 0x7f3864103350 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3863fff640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 0x7f3864197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3869215640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f386419efd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3854009230 tx=0x7f3854005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3861ffb640 1 -- 172.21.15.38:0/3156593641 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854041070 con 0x7f3864103350 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3861ffb640 1 -- 172.21.15.38:0/3156593641 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38540054e0 con 0x7f3864103350 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386419f8d0 con 0x7f3864103350 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f3861ffb640 1 -- 172.21.15.38:0/3156593641 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854034440 con 0x7f3864103350 2024-04-22T01:12:32.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.257+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386419fdc0 con 0x7f3864103350 2024-04-22T01:12:32.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.259+0000 7f3861ffb640 1 -- 172.21.15.38:0/3156593641 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3854044070 con 0x7f3864103350 2024-04-22T01:12:32.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 Processor -- start 2024-04-22T01:12:32.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 -- start start 2024-04-22T01:12:32.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d3177e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31783970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d31786470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3861ffb640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f384c0a2b90 0x7f384c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d316dedf0 con 0x558d3177e480 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558d31687640 con 0x558d31783fe0 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f386d85b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d316dadb0 con 0x558d3177ee70 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d3177e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3869215640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f384c0a2b90 0x7f384c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d3177e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49916/0 (socket says 172.21.15.38:49916) 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 -- 172.21.15.38:0/2267293413 learned_addr learned my addr 172.21.15.38:0/2267293413 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3861ffb640 1 -- 172.21.15.38:0/3156593641 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f38540f1280 con 0x7f3864103350 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3868a14640 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31783970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d31786470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 -- 172.21.15.38:0/2267293413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 msgr2=0x558d31783970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31783970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 -- 172.21.15.38:0/2267293413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 msgr2=0x558d31786470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d31786470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.260+0000 7f3863fff640 1 -- 172.21.15.38:0/2267293413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d313f32d0 con 0x558d3177e480 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f3869215640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f384c0a2b90 0x7f384c0a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f38540bd660 tx=0x7f3854090790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f3863fff640 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d3177e8a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3858005850 tx=0x7f385800b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b21c71f60c3d76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f384affd640 1 -- 172.21.15.38:0/2267293413 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858005220 con 0x558d3177e480 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f384affd640 1 -- 172.21.15.38:0/2267293413 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858004c80 con 0x558d3177e480 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f384affd640 1 -- 172.21.15.38:0/2267293413 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858012a80 con 0x558d3177e480 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 -- 172.21.15.38:0/2267293413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 msgr2=0x558d3177e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d3177e8a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3858005850 tx=0x7f385800b580 comp rx=0 tx=0).stop 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 -- 172.21.15.38:0/2267293413 shutdown_connections 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d31786470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31783970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 --2- 172.21.15.38:0/2267293413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d3177e8a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 -- 172.21.15.38:0/2267293413 >> 172.21.15.38:0/2267293413 conn(0x558d317755e0 msgr2=0x558d317759e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.261+0000 7f386d85b740 1 -- 172.21.15.38:0/2267293413 shutdown_connections 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 -- 172.21.15.38:0/2267293413 wait complete. 2024-04-22T01:12:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 -- start start 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 Processor -- start 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d31788a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31788f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d3178ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d31786f90 con 0x558d3177e480 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558d3169abe0 con 0x558d31783fe0 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d31787110 con 0x558d3177ee70 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f3863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d31788a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f386d85b740 10 client.0 _ll_register_callbacks cb 0x558d3169a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f3863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d31788a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49924/0 (socket says 172.21.15.38:49924) 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f3863fff640 1 -- 172.21.15.38:0/1232066296 learned_addr learned my addr 172.21.15.38:0/1232066296 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.262+0000 7f3868a14640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31788f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3863fff640 1 -- 172.21.15.38:0/1232066296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 msgr2=0x558d31788f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3863fff640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31788f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3863fff640 1 -- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 msgr2=0x558d3178ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3863fff640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d3178ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3863fff640 1 -- 172.21.15.38:0/1232066296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d314bd6b0 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3868a14640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31788f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3863fff640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d31788a10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3858005760 tx=0x7f3858005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0bcfdb84ca9ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3843fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3843fff640 1 -- 172.21.15.38:0/1232066296 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385802a060 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3843fff640 1 -- 172.21.15.38:0/1232066296 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f3858012800 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3858004c80 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3843fff640 1 -- 172.21.15.38:0/1232066296 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f385800b940 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f386d85b740 10 client.15273 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d31792f70 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3843fff640 1 -- 172.21.15.38:0/1232066296 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858004c80 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.263+0000 7f3843fff640 1 -- 172.21.15.38:0/1232066296 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3858025940 con 0x558d3177e480 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f386d85b740 10 client.15273 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f386d85b740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T01:12:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f386d85b740 20 client.15273 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f386d85b740 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d317951d0 0x558d31796640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f386d85b740 4 client.15273 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d316ec330 con 0x558d317951d0 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f3868a14640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d317951d0 0x558d31796640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f3843fff640 10 client.15273 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.264+0000 7f3868a14640 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d317951d0 0x558d31796640 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27cac823c508b0e0 server_cookie=750437f7fbddd5f6 in_seq=0 out_seq=0 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f3843fff640 1 -- 172.21.15.38:0/1232066296 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d316ec330 con 0x558d317951d0 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f3843fff640 10 client.15273 handle_command_reply: tid=0 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 client.15273 shutdown 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 msgr2=0x558d31788a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d31788a10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3858005760 tx=0x7f3858005250 comp rx=0 tx=0).stop 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 shutdown_connections 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d317951d0 0x558d31796640 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d31783fe0 0x558d3178ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d3177ee70 0x558d31788f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f3843fff640 0 client.15273 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 --2- 172.21.15.38:0/1232066296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d31788a10 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 >> 172.21.15.38:0/1232066296 conn(0x558d317755e0 msgr2=0x558d31774720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 shutdown_connections 2024-04-22T01:12:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.265+0000 7f386d85b740 1 -- 172.21.15.38:0/1232066296 wait complete. 2024-04-22T01:12:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.266+0000 7f386d85b740 20 client.15273 trim_cache size 0 max 16384 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 Processor -- start 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 -- start start 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d317759c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177ee70 0x558d3178ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d31788620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d31794980 con 0x558d3177e480 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558d31795050 con 0x558d3177ee70 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f386d85b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d31795720 con 0x558d31783fe0 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d317759c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d317759c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49932/0 (socket says 172.21.15.38:49932) 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 -- 172.21.15.38:0/1797562789 learned_addr learned my addr 172.21.15.38:0/1797562789 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 -- 172.21.15.38:0/1797562789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 msgr2=0x558d31788620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d31788620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3868a14640 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177ee70 0x558d3178ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 -- 172.21.15.38:0/1797562789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177ee70 msgr2=0x558d3178ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177ee70 0x558d3178ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 -- 172.21.15.38:0/1797562789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d31789a00 con 0x558d3177e480 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3868a14640 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177ee70 0x558d3178ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f3863fff640 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d317759c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3858004ff0 tx=0x7f3858004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e055ae16b0f10b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.284+0000 7f384a7fc640 1 -- 172.21.15.38:0/1797562789 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385802a030 con 0x558d3177e480 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f384a7fc640 1 -- 172.21.15.38:0/1797562789 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858004c80 con 0x558d3177e480 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f384a7fc640 1 -- 172.21.15.38:0/1797562789 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858025470 con 0x558d3177e480 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 -- 172.21.15.38:0/1797562789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 msgr2=0x558d317759c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d317759c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3858004ff0 tx=0x7f3858004a60 comp rx=0 tx=0).stop 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 -- 172.21.15.38:0/1797562789 shutdown_connections 2024-04-22T01:12:32.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d31788620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177ee70 0x558d3178ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 --2- 172.21.15.38:0/1797562789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177e480 0x558d317759c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 -- 172.21.15.38:0/1797562789 >> 172.21.15.38:0/1797562789 conn(0x558d317755e0 msgr2=0x558d3177f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 -- 172.21.15.38:0/1797562789 shutdown_connections 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 -- 172.21.15.38:0/1797562789 wait complete. 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 -- start start 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.285+0000 7f386d85b740 1 Processor -- start 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177e480 0x558d3178c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 0x558d3178c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d3178f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d317964c0 con 0x558d3177ee70 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558d31795df0 con 0x558d3177e480 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d31796b90 con 0x558d31783fe0 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f386d85b740 10 client.0 _ll_register_callbacks cb 0x7f385800b330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 0x558d3178c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 0x558d3178c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49938/0 (socket says 172.21.15.38:49938) 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 -- 172.21.15.38:0/2177693803 learned_addr learned my addr 172.21.15.38:0/2177693803 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3863fff640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d3178f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3863fff640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177e480 0x558d3178c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 -- 172.21.15.38:0/2177693803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 msgr2=0x558d3178f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d3178f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 -- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177e480 msgr2=0x558d3178c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177e480 0x558d3178c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 -- 172.21.15.38:0/2177693803 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d3178f8c0 con 0x558d3177ee70 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3863fff640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d3178f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3868a14640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 0x558d3178c6c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x558d3177f8e0 tx=0x7f385c009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6ceecf35b1f7c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3843fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.286+0000 7f3843fff640 1 -- 172.21.15.38:0/2177693803 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c012c70 con 0x558d3177ee70 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f3843fff640 1 -- 172.21.15.38:0/2177693803 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f385c01a2c0 con 0x558d3177ee70 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3858004c80 con 0x558d3177ee70 2024-04-22T01:12:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f3843fff640 1 -- 172.21.15.38:0/2177693803 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f385c02b050 con 0x558d3177ee70 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 10 client.15279 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d317fb7b0 con 0x558d3177ee70 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f3843fff640 1 -- 172.21.15.38:0/2177693803 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858004c80 con 0x558d3177ee70 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f3843fff640 1 -- 172.21.15.38:0/2177693803 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f385c00f040 con 0x558d3177ee70 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 10 client.15279 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 10 client.15279 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 20 client.15279 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d31800420 0x558d317ff130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 4 client.15279 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "status"}) -- 0x558d316e6680 con 0x558d31800420 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.287+0000 7f3863fff640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d31800420 0x558d317ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f3863fff640 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d31800420 0x558d317ff130 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9853a52355b14e8e server_cookie=92ac555a95805da7 in_seq=0 out_seq=0 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f3843fff640 10 client.15279 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f3843fff640 1 -- 172.21.15.38:0/2177693803 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x558d316e6680 con 0x558d31800420 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f3843fff640 10 client.15279 handle_command_reply: tid=0 2024-04-22T01:12:32.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 client.15279 shutdown 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 msgr2=0x558d3178c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 0x558d3178c6c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x558d3177f8e0 tx=0x7f385c009de0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 shutdown_connections 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x558d31800420 0x558d317ff130 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d31783fe0 0x558d3178f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d3177ee70 0x558d3178c6c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f3843fff640 0 client.15279 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 --2- 172.21.15.38:0/2177693803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d3177e480 0x558d3178c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 >> 172.21.15.38:0/2177693803 conn(0x558d317755e0 msgr2=0x558d317759c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 shutdown_connections 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 1 -- 172.21.15.38:0/2177693803 wait complete. 2024-04-22T01:12:32.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.288+0000 7f386d85b740 20 client.15279 trim_cache size 0 max 16384 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24527, 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 121.695633796, 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 125.854923247 2024-04-22T01:12:32.295 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:32.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f384c0a2b90 msgr2=0x7f384c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f384c0a2b90 0x7f384c0a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f38540bd660 tx=0x7f3854090790 comp rx=0 tx=0).stop 2024-04-22T01:12:32.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 msgr2=0x7f386419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:32.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f386419efd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3854009230 tx=0x7f3854005dd0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 shutdown_connections 2024-04-22T01:12:32.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f384c0a2b90 0x7f384c0a5050 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3864103350 0x7f386419efd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3864102990 0x7f3864197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 --2- 172.21.15.38:0/3156593641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38641017a0 0x7f3864197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:32.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 >> 172.21.15.38:0/3156593641 conn(0x7f38640fcd10 msgr2=0x7f3864105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:32.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 shutdown_connections 2024-04-22T01:12:32.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:32.289+0000 7f386ac78640 1 -- 172.21.15.38:0/3156593641 wait complete. 2024-04-22T01:12:32.300 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T01:12:32.300 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T01:12:32.310 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:12:32.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T01:12:32.556 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:33.688 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:33 smithi038 ceph-mon[28043]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s; 55 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T01:12:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:33 smithi111 ceph-mon[31057]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s; 55 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T01:12:34.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:33 smithi203 ceph-mon[31825]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s; 55 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.285+0000 7fbe746dc700 1 -- 172.21.15.38:0/1896984870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c101450 msgr2=0x7fbe6c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.285+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1896984870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c101450 0x7fbe6c1018d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009990 tx=0x7fbe68009ca0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 -- 172.21.15.38:0/1896984870 shutdown_connections 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1896984870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe6c101e10 0x7fbe6c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1896984870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1896984870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c101450 0x7fbe6c1018d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 -- 172.21.15.38:0/1896984870 >> 172.21.15.38:0/1896984870 conn(0x7fbe6c0fbea0 msgr2=0x7fbe6c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 -- 172.21.15.38:0/1896984870 shutdown_connections 2024-04-22T01:12:34.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 -- 172.21.15.38:0/1896984870 wait complete. 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 Processor -- start 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 -- start start 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.286+0000 7fbe746dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe6c101450 0x7fbe6c1051f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe746dc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe6c101e10 0x7fbe6c103840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe746dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c103d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe746dc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c10cde0 con 0x7fbe6c1027a0 2024-04-22T01:12:34.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe746dc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c10cc60 con 0x7fbe6c101450 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe746dc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c10cf60 con 0x7fbe6c101e10 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c103d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c103d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49948/0 (socket says 172.21.15.38:49948) 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 -- 172.21.15.38:0/1205540172 learned_addr learned my addr 172.21.15.38:0/1205540172 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72478700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe6c101450 0x7fbe6c1051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe71c77700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe6c101e10 0x7fbe6c103840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 -- 172.21.15.38:0/1205540172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe6c101e10 msgr2=0x7fbe6c103840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe6c101e10 0x7fbe6c103840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 -- 172.21.15.38:0/1205540172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe6c101450 msgr2=0x7fbe6c1051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe6c101450 0x7fbe6c1051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 -- 172.21.15.38:0/1205540172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6c104630 con 0x7fbe6c1027a0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe72c79700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c103d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbe64007ee0 tx=0x7fbe6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6400dbd0 con 0x7fbe6c1027a0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.287+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6400dd70 con 0x7fbe6c1027a0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.288+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6c1a2680 con 0x7fbe6c1027a0 2024-04-22T01:12:34.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.288+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64005a60 con 0x7fbe6c1027a0 2024-04-22T01:12:34.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.288+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6c1a2b90 con 0x7fbe6c1027a0 2024-04-22T01:12:34.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.289+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6c068f20 con 0x7fbe6c1027a0 2024-04-22T01:12:34.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.290+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbe6401a020 con 0x7fbe6c1027a0 2024-04-22T01:12:34.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.290+0000 7fbe5f7fe700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbe580a2ca0 0x7fbe580a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:34.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.290+0000 7fbe72478700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbe580a2ca0 0x7fbe580a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:34.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.290+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fbe640ceaf0 con 0x7fbe6c1027a0 2024-04-22T01:12:34.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.291+0000 7fbe72478700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbe580a2ca0 0x7fbe580a5160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c06a780 tx=0x7fbe6000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:34.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.295+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbe64010070 con 0x7fbe6c1027a0 2024-04-22T01:12:34.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.521+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fbe6c0fe000 con 0x7fbe580a2ca0 2024-04-22T01:12:34.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.523+0000 7fbe5f7fe700 1 -- 172.21.15.38:0/1205540172 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fbe6c0fe000 con 0x7fbe580a2ca0 2024-04-22T01:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout:Backend: cephadm 2024-04-22T01:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout:Available: Yes 2024-04-22T01:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout:Paused: No 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbe580a2ca0 msgr2=0x7fbe580a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbe580a2ca0 0x7fbe580a5160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c06a780 tx=0x7fbe6000b410 comp rx=0 tx=0).stop 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 msgr2=0x7fbe6c103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c103d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbe64007ee0 tx=0x7fbe6400d3b0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 shutdown_connections 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fbe580a2ca0 0x7fbe580a5160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe6c1027a0 0x7fbe6c103d80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe6c101450 0x7fbe6c1051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 --2- 172.21.15.38:0/1205540172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe6c101e10 0x7fbe6c103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 >> 172.21.15.38:0/1205540172 conn(0x7fbe6c0fbea0 msgr2=0x7fbe6c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 shutdown_connections 2024-04-22T01:12:34.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:34.525+0000 7fbe746dc700 1 -- 172.21.15.38:0/1205540172 wait complete. 2024-04-22T01:12:35.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T01:12:35.411 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:35.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:35 smithi038 ceph-mon[28043]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T01:12:35.820 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:35 smithi038 ceph-mon[28043]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:35 smithi111 ceph-mon[31057]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T01:12:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:35 smithi111 ceph-mon[31057]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:35 smithi203 ceph-mon[31825]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T01:12:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:35 smithi203 ceph-mon[31825]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3234951879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e68103e60 msgr2=0x7f1e6810a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3234951879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e68103e60 0x7f1e6810a710 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000b0d0 tx=0x7f1e6000b3e0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3234951879 shutdown_connections 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3234951879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e68103e60 0x7f1e6810a710 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3234951879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e681022c0 0x7f1e681026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3234951879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 0x7f1e68103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3234951879 >> 172.21.15.38:0/3234951879 conn(0x7f1e680fd880 msgr2=0x7f1e680ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3234951879 shutdown_connections 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.150+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3234951879 wait complete. 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 Processor -- start 2024-04-22T01:12:37.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 -- start start 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 0x7f1e68198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 0x7f1e68198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68103e60 0x7f1e6819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6810df00 con 0x7f1e681022c0 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6810dd80 con 0x7f1e68103e60 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6ff5f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6810e080 con 0x7f1e681034c0 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6dcfb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 0x7f1e68198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6dcfb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 0x7f1e68198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35232/0 (socket says 172.21.15.38:35232) 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6dcfb700 1 -- 172.21.15.38:0/3185026011 learned_addr learned my addr 172.21.15.38:0/3185026011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6d4fa700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 0x7f1e68198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.151+0000 7f1e6e4fc700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68103e60 0x7f1e6819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6dcfb700 1 -- 172.21.15.38:0/3185026011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 msgr2=0x7f1e68198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6dcfb700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 0x7f1e68198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6dcfb700 1 -- 172.21.15.38:0/3185026011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68103e60 msgr2=0x7f1e6819fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6dcfb700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68103e60 0x7f1e6819fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6dcfb700 1 -- 172.21.15.38:0/3185026011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e681a00b0 con 0x7f1e681022c0 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6d4fa700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 0x7f1e68198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6dcfb700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 0x7f1e68198450 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1e5800c9a0 tx=0x7f1e5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e58004220 con 0x7f1e681022c0 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e580043c0 con 0x7f1e681022c0 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e681a03a0 con 0x7f1e681022c0 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e58003c90 con 0x7f1e681022c0 2024-04-22T01:12:37.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.152+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e681a08e0 con 0x7f1e681022c0 2024-04-22T01:12:37.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.153+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e68068f20 con 0x7f1e681022c0 2024-04-22T01:12:37.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.154+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1e580076d0 con 0x7f1e681022c0 2024-04-22T01:12:37.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.155+0000 7f1e5effd700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e540a2c20 0x7f1e540a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:37.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.155+0000 7f1e6d4fa700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e540a2c20 0x7f1e540a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:37.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.155+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f1e580cf3c0 con 0x7f1e681022c0 2024-04-22T01:12:37.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.155+0000 7f1e6d4fa700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e540a2c20 0x7f1e540a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1e640104e0 tx=0x7f1e6400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:37.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.158+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e58014030 con 0x7f1e681022c0 2024-04-22T01:12:37.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.374+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f1e68062670 con 0x7f1e540a2c20 2024-04-22T01:12:37.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.390+0000 7f1e5effd700 1 -- 172.21.15.38:0/3185026011 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f1e68062670 con 0x7f1e540a2c20 2024-04-22T01:12:37.390 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.a smithi038 running (2m) 31s ago 2m 16.0M - 19.0.0-2405-g820a2f8c 6702148920ad cccf62f2b37b 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.b smithi111 running (2m) 64s ago 2m 26.2M - 19.0.0-2405-g820a2f8c 6702148920ad 4fee09027b0a 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.c smithi203 running (2m) 65s ago 2m 16.1M - 19.0.0-2405-g820a2f8c 6702148920ad a2804210d067 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.d smithi038 running (2m) 31s ago 2m 14.1M - 19.0.0-2405-g820a2f8c 6702148920ad f1bf8db94a19 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.e smithi111 running (2m) 64s ago 2m 16.7M - 19.0.0-2405-g820a2f8c 6702148920ad cb02eaa3192a 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.f smithi203 running (2m) 65s ago 2m 16.7M - 19.0.0-2405-g820a2f8c 6702148920ad aaf036dce494 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.g smithi038 running (2m) 31s ago 2m 16.2M - 19.0.0-2405-g820a2f8c 6702148920ad fc39d84caa4e 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.h smithi111 running (2m) 64s ago 2m 16.6M - 19.0.0-2405-g820a2f8c 6702148920ad a7209b95fa17 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.i smithi203 running (2m) 65s ago 2m 16.0M - 19.0.0-2405-g820a2f8c 6702148920ad b20e95da79c9 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.j smithi038 running (2m) 31s ago 2m 15.5M - 19.0.0-2405-g820a2f8c 6702148920ad e0f2e6ad64c3 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.k smithi111 running (2m) 64s ago 2m 17.4M - 19.0.0-2405-g820a2f8c 6702148920ad f82f4e5060e9 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mds.l smithi203 running (2m) 65s ago 2m 17.4M - 19.0.0-2405-g820a2f8c 6702148920ad e0d05dfe669b 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.x smithi038 *:9283,8765 running (10m) 31s ago 10m 515M - 19.0.0-2405-g820a2f8c 6702148920ad 4798ca6bfa46 2024-04-22T01:12:37.391 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.y smithi111 *:8443,8765 running (8m) 64s ago 8m 452M - 19.0.0-2405-g820a2f8c 6702148920ad 360ce42af3ec 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.z smithi203 *:8443,8765 running (8m) 65s ago 8m 451M - 19.0.0-2405-g820a2f8c 6702148920ad 3277d117afd7 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a smithi038 running (10m) 31s ago 10m 50.8M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 265dc448bbc4 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b smithi111 running (8m) 64s ago 8m 40.2M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 1e22d75ebb5d 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c smithi203 running (8m) 65s ago 8m 40.3M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 11b7a1349670 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (7m) 31s ago 7m 46.0M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 7d0e45e7a28e 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi111 running (7m) 64s ago 7m 46.8M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 33adae90a82a 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi203 running (7m) 65s ago 7m 57.1M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 9799ae04b533 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi038 running (6m) 31s ago 6m 57.5M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 6bb333755e2f 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi111 running (6m) 64s ago 6m 46.0M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 650a73fe5049 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi203 running (5m) 65s ago 5m 44.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 4aebeda6765e 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 smithi038 running (5m) 31s ago 5m 57.2M 4273M 19.0.0-2405-g820a2f8c 6702148920ad f759c73fac75 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 smithi111 running (4m) 64s ago 4m 44.0M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 026b18b63421 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 smithi203 running (4m) 65s ago 4m 54.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad b52cf94bc720 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 smithi038 running (3m) 31s ago 3m 42.2M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 4cbf76feea3d 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 smithi111 running (3m) 64s ago 3m 43.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 8eb9d401545f 2024-04-22T01:12:37.392 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 smithi203 running (3m) 65s ago 3m 52.4M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 9896b87a5854 2024-04-22T01:12:37.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e540a2c20 msgr2=0x7f1e540a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:37.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e540a2c20 0x7f1e540a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1e640104e0 tx=0x7f1e6400b3c0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 msgr2=0x7f1e68198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:37.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 0x7f1e68198450 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1e5800c9a0 tx=0x7f1e5800ccb0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 shutdown_connections 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e540a2c20 0x7f1e540a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e68103e60 0x7f1e6819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e681022c0 0x7f1e68198450 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 --2- 172.21.15.38:0/3185026011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e681034c0 0x7f1e68198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.392+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 >> 172.21.15.38:0/3185026011 conn(0x7f1e680fd880 msgr2=0x7f1e681066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.393+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 shutdown_connections 2024-04-22T01:12:37.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:37.393+0000 7f1e6ff5f700 1 -- 172.21.15.38:0/3185026011 wait complete. 2024-04-22T01:12:37.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:37 smithi038 ceph-mon[28043]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:37 smithi111 ceph-mon[31057]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:37 smithi203 ceph-mon[31825]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:38.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T01:12:38.301 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:38.676 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:38 smithi038 ceph-mon[28043]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:38.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:38 smithi111 ceph-mon[31057]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:39.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:38 smithi203 ceph-mon[31825]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:39.720 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:39 smithi038 ceph-mon[28043]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:39 smithi111 ceph-mon[31057]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 -- 172.21.15.38:0/925194103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110102290 msgr2=0x7f91101026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 --2- 172.21.15.38:0/925194103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110102290 0x7f91101026b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f91040099c0 tx=0x7f9104009cd0 comp rx=0 tx=0).stop 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 -- 172.21.15.38:0/925194103 shutdown_connections 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 --2- 172.21.15.38:0/925194103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103e30 0x7f911010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 --2- 172.21.15.38:0/925194103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110102290 0x7f91101026b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 --2- 172.21.15.38:0/925194103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9110103490 0x7f91101038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 -- 172.21.15.38:0/925194103 >> 172.21.15.38:0/925194103 conn(0x7f91100fd830 msgr2=0x7f91100ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 -- 172.21.15.38:0/925194103 shutdown_connections 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.946+0000 7f9116c1e700 1 -- 172.21.15.38:0/925194103 wait complete. 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 Processor -- start 2024-04-22T01:12:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 -- start start 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9110102290 0x7f9110075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103490 0x7f9110074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 0x7f9110074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f911010df50 con 0x7f9110103e30 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f911010ddd0 con 0x7f9110103490 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f9116c1e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f911010e0d0 con 0x7f9110102290 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f91151bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 0x7f9110074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f91151bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 0x7f9110074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35246/0 (socket says 172.21.15.38:35246) 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f91151bb700 1 -- 172.21.15.38:0/1624845882 learned_addr learned my addr 172.21.15.38:0/1624845882 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.947+0000 7f91149ba700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9110102290 0x7f9110075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f910ffff700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103490 0x7f9110074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f91151bb700 1 -- 172.21.15.38:0/1624845882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9110102290 msgr2=0x7f9110075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f91151bb700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9110102290 0x7f9110075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:39.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f91151bb700 1 -- 172.21.15.38:0/1624845882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103490 msgr2=0x7f9110074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f91151bb700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103490 0x7f9110074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f91151bb700 1 -- 172.21.15.38:0/1624845882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9110075410 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f910ffff700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103490 0x7f9110074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f91151bb700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 0x7f9110074b60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f910800b970 tx=0x7f910800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9108019070 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f910800ce10 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91101a27e0 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9108007370 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.948+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91101a2d70 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.949+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9110068f20 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.950+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f910801c070 con 0x7f9110103e30 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.951+0000 7f910dffb700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91000a2ca0 0x7f91000a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.951+0000 7f91149ba700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91000a2ca0 0x7f91000a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:39.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.951+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f91080ce770 con 0x7f9110103e30 2024-04-22T01:12:39.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.952+0000 7f91149ba700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91000a2ca0 0x7f91000a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9104003890 tx=0x7f9104005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:39.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:39.956+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f910809b300 con 0x7f9110103e30 2024-04-22T01:12:40.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:39 smithi203 ceph-mon[31825]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:40.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.171+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f9110062670 con 0x7f91000a2ca0 2024-04-22T01:12:40.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.177+0000 7f910dffb700 1 -- 172.21.15.38:0/1624845882 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f9110062670 con 0x7f91000a2ca0 2024-04-22T01:12:40.177 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T01:12:40.178 INFO:teuthology.orchestra.run.smithi038.stdout:mds.all 12/12 67s ago 2m smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-22T01:12:40.178 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 3/3 67s ago 8m smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-22T01:12:40.178 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 67s ago 8m smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c;count:3 2024-04-22T01:12:40.178 INFO:teuthology.orchestra.run.smithi038.stdout:osd 12 67s ago - 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91000a2ca0 msgr2=0x7f91000a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91000a2ca0 0x7f91000a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9104003890 tx=0x7f9104005fb0 comp rx=0 tx=0).stop 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 msgr2=0x7f9110074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 0x7f9110074b60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f910800b970 tx=0x7f910800bc80 comp rx=0 tx=0).stop 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 shutdown_connections 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f91000a2ca0 0x7f91000a5160 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9110103e30 0x7f9110074b60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9110102290 0x7f9110075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 --2- 172.21.15.38:0/1624845882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9110103490 0x7f9110074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:40.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 >> 172.21.15.38:0/1624845882 conn(0x7f91100fd830 msgr2=0x7f91101068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:40.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 shutdown_connections 2024-04-22T01:12:40.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:40.179+0000 7f9116c1e700 1 -- 172.21.15.38:0/1624845882 wait complete. 2024-04-22T01:12:40.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T01:12:41.088 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:41 smithi111 ceph-mon[31057]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:41 smithi111 ceph-mon[31057]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:41 smithi038 ceph-mon[28043]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:41.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:41 smithi038 ceph-mon[28043]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:42.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:41 smithi203 ceph-mon[31825]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:12:42.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:41 smithi203 ceph-mon[31825]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 -- 172.21.15.38:0/2290333102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 msgr2=0x7fe0d81051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/2290333102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81051b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009990 tx=0x7fe0d4009ca0 comp rx=0 tx=0).stop 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 -- 172.21.15.38:0/2290333102 shutdown_connections 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/2290333102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0d8108020 0x7fe0d810a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/2290333102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81051b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/2290333102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0d81056f0 0x7fe0d8107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:42.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 -- 172.21.15.38:0/2290333102 >> 172.21.15.38:0/2290333102 conn(0x7fe0d80fc730 msgr2=0x7fe0d80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:42.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.903+0000 7fe0e07b7700 1 -- 172.21.15.38:0/2290333102 shutdown_connections 2024-04-22T01:12:42.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 -- 172.21.15.38:0/2290333102 wait complete. 2024-04-22T01:12:42.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 Processor -- start 2024-04-22T01:12:42.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 -- start start 2024-04-22T01:12:42.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:42.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0d81056f0 0x7fe0d8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:42.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0d8108020 0x7fe0d819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.904+0000 7fe0e07b7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d810df10 con 0x7fe0d8102dc0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0e07b7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d810dd90 con 0x7fe0d8108020 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0e07b7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d810e090 con 0x7fe0d81056f0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35270/0 (socket says 172.21.15.38:35270) 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 -- 172.21.15.38:0/4207885249 learned_addr learned my addr 172.21.15.38:0/4207885249 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0ded54700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0d8108020 0x7fe0d819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0ddd52700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0d81056f0 0x7fe0d8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 -- 172.21.15.38:0/4207885249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0d81056f0 msgr2=0x7fe0d8198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0d81056f0 0x7fe0d8198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 -- 172.21.15.38:0/4207885249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0d8108020 msgr2=0x7fe0d819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0d8108020 0x7fe0d819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 -- 172.21.15.38:0/4207885249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0d81a0240 con 0x7fe0d8102dc0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0de553700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81985e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005910 tx=0x7fe0d4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:42.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4026070 con 0x7fe0d8102dc0 2024-04-22T01:12:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0d81a04d0 con 0x7fe0d8102dc0 2024-04-22T01:12:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.905+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0d4005510 con 0x7fe0d8102dc0 2024-04-22T01:12:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.906+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4003cc0 con 0x7fe0d8102dc0 2024-04-22T01:12:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.906+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0d81a0a00 con 0x7fe0d8102dc0 2024-04-22T01:12:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.907+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0d8068f20 con 0x7fe0d8102dc0 2024-04-22T01:12:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.908+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe0d4029070 con 0x7fe0d8102dc0 2024-04-22T01:12:42.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.908+0000 7fe0cf7fe700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe0c40a2ca0 0x7fe0c40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:42.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.909+0000 7fe0ddd52700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe0c40a2ca0 0x7fe0c40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:42.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.909+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe0d40cfc90 con 0x7fe0d8102dc0 2024-04-22T01:12:42.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.909+0000 7fe0ddd52700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe0c40a2ca0 0x7fe0c40a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009a10 tx=0x7fe0c8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:42.913+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe0d405d6f0 con 0x7fe0d8102dc0 2024-04-22T01:12:43.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.125+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe0d8062670 con 0x7fe0c40a2ca0 2024-04-22T01:12:43.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.126+0000 7fe0cf7fe700 1 -- 172.21.15.38:0/4207885249 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fe0d8062670 con 0x7fe0c40a2ca0 2024-04-22T01:12:43.126 INFO:teuthology.orchestra.run.smithi038.stdout:HOST ADDR LABELS STATUS 2024-04-22T01:12:43.126 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 172.21.15.38 2024-04-22T01:12:43.127 INFO:teuthology.orchestra.run.smithi038.stdout:smithi111 172.21.15.111 2024-04-22T01:12:43.127 INFO:teuthology.orchestra.run.smithi038.stdout:smithi203 172.21.15.203 2024-04-22T01:12:43.127 INFO:teuthology.orchestra.run.smithi038.stdout:3 hosts in cluster 2024-04-22T01:12:43.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe0c40a2ca0 msgr2=0x7fe0c40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe0c40a2ca0 0x7fe0c40a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009a10 tx=0x7fe0c8009190 comp rx=0 tx=0).stop 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 msgr2=0x7fe0d81985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81985e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005910 tx=0x7fe0d4009580 comp rx=0 tx=0).stop 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 shutdown_connections 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe0c40a2ca0 0x7fe0c40a5160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0d8108020 0x7fe0d819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0d8102dc0 0x7fe0d81985e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 --2- 172.21.15.38:0/4207885249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0d81056f0 0x7fe0d8198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 >> 172.21.15.38:0/4207885249 conn(0x7fe0d80fc730 msgr2=0x7fe0d8103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 shutdown_connections 2024-04-22T01:12:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:43.128+0000 7fe0e07b7700 1 -- 172.21.15.38:0/4207885249 wait complete. 2024-04-22T01:12:43.767 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:43 smithi038 ceph-mon[28043]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T01:12:43.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T01:12:43.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:43 smithi111 ceph-mon[31057]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T01:12:44.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:43 smithi203 ceph-mon[31825]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T01:12:44.051 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:44.890 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:44 smithi038 ceph-mon[28043]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:44.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:44 smithi111 ceph-mon[31057]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:45.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:44 smithi203 ceph-mon[31825]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:45.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 -- 172.21.15.38:0/705795740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 msgr2=0x7fa8541015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:45.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 --2- 172.21.15.38:0/705795740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa8541015b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa8500099c0 tx=0x7fa850009cd0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 -- 172.21.15.38:0/705795740 shutdown_connections 2024-04-22T01:12:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 --2- 172.21.15.38:0/705795740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa854102d30 0x7fa8541095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 --2- 172.21.15.38:0/705795740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa8541015b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 --2- 172.21.15.38:0/705795740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa854102390 0x7fa8541027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 -- 172.21.15.38:0/705795740 >> 172.21.15.38:0/705795740 conn(0x7fa8540fc750 msgr2=0x7fa8540feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 -- 172.21.15.38:0/705795740 shutdown_connections 2024-04-22T01:12:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.743+0000 7fa85bd95700 1 -- 172.21.15.38:0/705795740 wait complete. 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 Processor -- start 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 -- start start 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa854197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa854102390 0x7fa854197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa854102d30 0x7fa85419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa85410cde0 con 0x7fa854101190 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa85410cc60 con 0x7fa854102d30 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85bd95700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa85410cf60 con 0x7fa854102390 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa854197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa854197420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35294/0 (socket says 172.21.15.38:35294) 2024-04-22T01:12:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 -- 172.21.15.38:0/2036045130 learned_addr learned my addr 172.21.15.38:0/2036045130 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa85a332700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa854102d30 0x7fa85419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859330700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa854102390 0x7fa854197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 -- 172.21.15.38:0/2036045130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa854102390 msgr2=0x7fa854197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa854102390 0x7fa854197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 -- 172.21.15.38:0/2036045130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa854102d30 msgr2=0x7fa85419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa854102d30 0x7fa85419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.744+0000 7fa859b31700 1 -- 172.21.15.38:0/2036045130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85419f080 con 0x7fa854101190 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.745+0000 7fa859b31700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa854197420 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c010 tx=0x7fa850009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.745+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850027070 con 0x7fa854101190 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.745+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa850004150 con 0x7fa854101190 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.745+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85419f310 con 0x7fa854101190 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.745+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85000f460 con 0x7fa854101190 2024-04-22T01:12:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.745+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85419f7c0 con 0x7fa854101190 2024-04-22T01:12:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.746+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa854068f20 con 0x7fa854101190 2024-04-22T01:12:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.747+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa85002a070 con 0x7fa854101190 2024-04-22T01:12:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.748+0000 7fa84affd700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa8400a2ca0 0x7fa8400a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.748+0000 7fa859330700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa8400a2ca0 0x7fa8400a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.748+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa8500d08f0 con 0x7fa854101190 2024-04-22T01:12:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.748+0000 7fa859330700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa8400a2ca0 0x7fa8400a5160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa844005950 tx=0x7fa8440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:45.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.751+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa850031070 con 0x7fa854101190 2024-04-22T01:12:45.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:45 smithi111 ceph-mon[31057]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-22T01:12:45.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:45 smithi038 ceph-mon[28043]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-22T01:12:45.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.967+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fa854062670 con 0x7fa8400a2ca0 2024-04-22T01:12:45.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.969+0000 7fa84affd700 1 -- 172.21.15.38:0/2036045130 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fa854062670 con 0x7fa8400a2ca0 2024-04-22T01:12:45.969 INFO:teuthology.orchestra.run.smithi038.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T01:12:45.970 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-22T01:12:45.970 INFO:teuthology.orchestra.run.smithi038.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T01:12:45.970 INFO:teuthology.orchestra.run.smithi038.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa8400a2ca0 msgr2=0x7fa8400a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa8400a2ca0 0x7fa8400a5160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa844005950 tx=0x7fa8440058e0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 msgr2=0x7fa854197420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa854197420 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c010 tx=0x7fa850009580 comp rx=0 tx=0).stop 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 shutdown_connections 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa8400a2ca0 0x7fa8400a5160 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa854102d30 0x7fa85419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa854101190 0x7fa854197420 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 --2- 172.21.15.38:0/2036045130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa854102390 0x7fa854197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.971+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 >> 172.21.15.38:0/2036045130 conn(0x7fa8540fc750 msgr2=0x7fa8541056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.972+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 shutdown_connections 2024-04-22T01:12:45.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:45.972+0000 7fa85bd95700 1 -- 172.21.15.38:0/2036045130 wait complete. 2024-04-22T01:12:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:45 smithi203 ceph-mon[31825]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-22T01:12:46.709 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T01:12:46.718 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:12:46.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T01:12:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:47 smithi111 ceph-mon[31057]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-22T01:12:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:47 smithi111 ceph-mon[31057]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:47 smithi038 ceph-mon[28043]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-22T01:12:47.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:47 smithi038 ceph-mon[28043]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:48.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:47 smithi203 ceph-mon[31825]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-22T01:12:48.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:47 smithi203 ceph-mon[31825]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 -- 172.21.15.38:0/1843131641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103490 msgr2=0x7f50481038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 --2- 172.21.15.38:0/1843131641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103490 0x7f50481038f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f50380099c0 tx=0x7f5038009cd0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 -- 172.21.15.38:0/1843131641 shutdown_connections 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 --2- 172.21.15.38:0/1843131641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5048103e30 0x7f504810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 --2- 172.21.15.38:0/1843131641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5048102290 0x7f50481026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 --2- 172.21.15.38:0/1843131641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103490 0x7f50481038f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.637+0000 7f504d410700 1 -- 172.21.15.38:0/1843131641 >> 172.21.15.38:0/1843131641 conn(0x7f50480fd830 msgr2=0x7f50480ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.638+0000 7f504d410700 1 -- 172.21.15.38:0/1843131641 shutdown_connections 2024-04-22T01:12:48.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.638+0000 7f504d410700 1 -- 172.21.15.38:0/1843131641 wait complete. 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.638+0000 7f504d410700 1 Processor -- start 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.638+0000 7f504d410700 1 -- start start 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f504d410700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5048102290 0x7f5048075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f504d410700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5048103490 0x7f5048074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f504d410700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 0x7f5048074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f504d410700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f504810de70 con 0x7f5048103e30 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f504d410700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f504810dcf0 con 0x7f5048102290 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f504d410700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f504810dff0 con 0x7f5048103490 2024-04-22T01:12:48.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 0x7f5048074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 0x7f5048074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40968/0 (socket says 172.21.15.38:40968) 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 -- 172.21.15.38:0/785817160 learned_addr learned my addr 172.21.15.38:0/785817160 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f5046ffd700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5048102290 0x7f5048075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50467fc700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5048103490 0x7f5048074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 -- 172.21.15.38:0/785817160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5048103490 msgr2=0x7f5048074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5048103490 0x7f5048074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 -- 172.21.15.38:0/785817160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5048102290 msgr2=0x7f5048075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5048102290 0x7f5048075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 -- 172.21.15.38:0/785817160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5048075410 con 0x7f5048103e30 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.639+0000 7f50477fe700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 0x7f5048074b60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f504000ada0 tx=0x7f504000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.640+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040017020 con 0x7f5048103e30 2024-04-22T01:12:48.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.640+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f504000cdb0 con 0x7f5048103e30 2024-04-22T01:12:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.640+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50481a25f0 con 0x7f5048103e30 2024-04-22T01:12:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.640+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50400053e0 con 0x7f5048103e30 2024-04-22T01:12:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.640+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50481a2b00 con 0x7f5048103e30 2024-04-22T01:12:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.641+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5048068f20 con 0x7f5048103e30 2024-04-22T01:12:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.643+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f504000c8d0 con 0x7f5048103e30 2024-04-22T01:12:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.643+0000 7f503ffff700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50340a2c20 0x7f50340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.644+0000 7f5046ffd700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50340a2c20 0x7f50340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.644+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f50400cf110 con 0x7f5048103e30 2024-04-22T01:12:48.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.644+0000 7f5046ffd700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50340a2c20 0x7f50340a50e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f50481032f0 tx=0x7f503000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.647+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5040021020 con 0x7f5048103e30 2024-04-22T01:12:48.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.906+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f504802df30 con 0x7f5048103e30 2024-04-22T01:12:48.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.906+0000 7f503ffff700 1 -- 172.21.15.38:0/785817160 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3102 (secure 0 0 0) 0x7f504009bcf0 con 0x7f5048103e30 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:e27 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:legacy client fscid: 1 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem 'cephfs' (1) 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:fs_name cephfs 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 27 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-04-22T01:10:29.943110+0000 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-04-22T01:11:00.429607+0000 2024-04-22T01:12:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:tableserver 0 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:root 0 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:session_timeout 300 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:session_autoclose 300 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:max_file_size 1099511627776 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:max_xattr_size 65536 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:required_client_features {} 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure 0 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure_osd_epoch 0 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:max_mds 5 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:in 0,1,2,3,4 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:up {0=24527,1=24485,2=24472,3=24451,4=24446} 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:failed 2024-04-22T01:12:48.908 INFO:teuthology.orchestra.run.smithi038.stdout:damaged 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:stopped 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:data_pools [3,4] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:metadata_pool 2 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:inline_data disabled 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:balancer 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:bal_rank_mask -1 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:standby_count_wanted 0 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:qdb_cluster leader: 24527 members: 24451,24472,24446,24527,24485 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.b{0:24527} state up:active seq 3 addr [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.f{0:24439} state up:standby-replay seq 1 addr [v2:172.21.15.203:6834/4027190739,v1:172.21.15.203:6835/4027190739] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.h{1:24485} state up:active seq 12 addr [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.k{1:24413} state up:standby-replay seq 1 addr [v2:172.21.15.111:6832/1598672021,v1:172.21.15.111:6833/1598672021] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.i{2:24472} state up:active seq 12 addr [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.l{2:24400} state up:standby-replay seq 1 addr [v2:172.21.15.203:6832/3645664141,v1:172.21.15.203:6833/3645664141] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.909 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.c{3:24451} state up:active seq 13 addr [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.a{3:14607} state up:standby-replay seq 1 addr [v2:172.21.15.38:6840/3631879100,v1:172.21.15.38:6841/3631879100] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.e{4:24446} state up:active seq 15 addr [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.g{4:14595} state up:standby-replay seq 1 addr [v2:172.21.15.38:6838/1149927748,v1:172.21.15.38:6839/1149927748] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout:Standby daemons: 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.j{-1:14514} state up:standby seq 1 addr [v2:172.21.15.38:6834/2658040728,v1:172.21.15.38:6835/2658040728] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.910 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.d{-1:14532} state up:standby seq 1 addr [v2:172.21.15.38:6836/3419129141,v1:172.21.15.38:6837/3419129141] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50340a2c20 msgr2=0x7f50340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50340a2c20 0x7f50340a50e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f50481032f0 tx=0x7f503000a380 comp rx=0 tx=0).stop 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 msgr2=0x7f5048074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 0x7f5048074b60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f504000ada0 tx=0x7f504000c330 comp rx=0 tx=0).stop 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 shutdown_connections 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50340a2c20 0x7f50340a50e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5048103e30 0x7f5048074b60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5048102290 0x7f5048075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 --2- 172.21.15.38:0/785817160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5048103490 0x7f5048074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.908+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 >> 172.21.15.38:0/785817160 conn(0x7f50480fd830 msgr2=0x7f5048106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:48.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.909+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 shutdown_connections 2024-04-22T01:12:48.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:48.909+0000 7f504d410700 1 -- 172.21.15.38:0/785817160 wait complete. 2024-04-22T01:12:48.912 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:12:49.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-22T01:12:49.808 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:49 smithi038 ceph-mon[28043]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T01:12:49.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:49 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/785817160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T01:12:50.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:49 smithi203 ceph-mon[31825]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T01:12:50.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:49 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/785817160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T01:12:50.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:49 smithi111 ceph-mon[31057]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T01:12:50.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:49 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/785817160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T01:12:51.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 -- 172.21.15.38:0/2615716628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 msgr2=0x7efe54102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 --2- 172.21.15.38:0/2615716628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe54102330 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00b0a0 tx=0x7efe4c00b3b0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 -- 172.21.15.38:0/2615716628 shutdown_connections 2024-04-22T01:12:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 --2- 172.21.15.38:0/2615716628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe54102330 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 --2- 172.21.15.38:0/2615716628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 0x7efe54100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 --2- 172.21.15.38:0/2615716628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe541014f0 0x7efe54101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 -- 172.21.15.38:0/2615716628 >> 172.21.15.38:0/2615716628 conn(0x7efe540fc750 msgr2=0x7efe540feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 -- 172.21.15.38:0/2615716628 shutdown_connections 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 -- 172.21.15.38:0/2615716628 wait complete. 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.498+0000 7efe5944b700 1 Processor -- start 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 -- start start 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 0x7efe5419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe541014f0 0x7efe5419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe541a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efe54110070 con 0x7efe54101eb0 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efe5410fef0 con 0x7efe541014f0 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe5944b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efe541101f0 con 0x7efe54100b90 2024-04-22T01:12:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe52ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 0x7efe5419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe541a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe541a1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40986/0 (socket says 172.21.15.38:40986) 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe537fe700 1 -- 172.21.15.38:0/2060800960 learned_addr learned my addr 172.21.15.38:0/2060800960 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.499+0000 7efe527fc700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe541014f0 0x7efe5419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe537fe700 1 -- 172.21.15.38:0/2060800960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 msgr2=0x7efe5419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe537fe700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 0x7efe5419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe537fe700 1 -- 172.21.15.38:0/2060800960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe541014f0 msgr2=0x7efe5419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe537fe700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe541014f0 0x7efe5419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe537fe700 1 -- 172.21.15.38:0/2060800960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe541a2270 con 0x7efe54101eb0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe52ffd700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 0x7efe5419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe537fe700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe541a1bd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00d010 tx=0x7efe4c004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c021070 con 0x7efe54101eb0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe541a2500 con 0x7efe54101eb0 2024-04-22T01:12:51.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe4c007b80 con 0x7efe54101eb0 2024-04-22T01:12:51.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe541a2a00 con 0x7efe54101eb0 2024-04-22T01:12:51.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.500+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c01d450 con 0x7efe54101eb0 2024-04-22T01:12:51.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.501+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe54068f20 con 0x7efe54101eb0 2024-04-22T01:12:51.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.503+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efe4c004140 con 0x7efe54101eb0 2024-04-22T01:12:51.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.504+0000 7efe3bfff700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe3c0a2c20 0x7efe3c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:51.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.504+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7efe4c00f020 con 0x7efe54101eb0 2024-04-22T01:12:51.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.504+0000 7efe52ffd700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe3c0a2c20 0x7efe3c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:51.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.504+0000 7efe52ffd700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe3c0a2c20 0x7efe3c0a50e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe440104e0 tx=0x7efe4400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:51.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.507+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efe4c02a020 con 0x7efe54101eb0 2024-04-22T01:12:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.716+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7efe5402db30 con 0x7efe54101eb0 2024-04-22T01:12:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.717+0000 7efe3bfff700 1 -- 172.21.15.38:0/2060800960 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5719 (secure 0 0 0) 0x7efe4c09da20 con 0x7efe54101eb0 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 93 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:fsid caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-04-22T01:02:20.325840+0000 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-04-22T01:12:17.458713+0000 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 27 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2024-04-22T01:12:51.718 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client luminous 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 12 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6802/3120557304,v1:172.21.15.38:6803/3120557304] [v2:172.21.15.38:6804/3120557304,v1:172.21.15.38:6805/3120557304] exists,up c263cffd-6624-47e2-935b-671cc5cb79b2 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6800/3032046140,v1:172.21.15.111:6801/3032046140] [v2:172.21.15.111:6802/3032046140,v1:172.21.15.111:6803/3032046140] exists,up 01d9f61a-90f2-4b76-98b8-fe7e49315113 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/1524209627,v1:172.21.15.203:6801/1524209627] [v2:172.21.15.203:6802/1524209627,v1:172.21.15.203:6803/1524209627] exists,up 8ce3eb04-5c85-4391-92f6-dd636ae03c6e 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6810/3150547332,v1:172.21.15.38:6811/3150547332] [v2:172.21.15.38:6812/3150547332,v1:172.21.15.38:6813/3150547332] exists,up e29b0ec6-0914-46b2-9887-c808bd3ab6b2 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6808/2127500568,v1:172.21.15.111:6809/2127500568] [v2:172.21.15.111:6810/2127500568,v1:172.21.15.111:6811/2127500568] exists,up 05e84962-4508-4457-a9ea-c52fd31377b4 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/3881588829,v1:172.21.15.203:6809/3881588829] [v2:172.21.15.203:6810/3881588829,v1:172.21.15.203:6811/3881588829] exists,up 0e16f8a4-4174-4264-a7ff-976293ac88c1 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6818/3905664243,v1:172.21.15.38:6819/3905664243] [v2:172.21.15.38:6820/3905664243,v1:172.21.15.38:6821/3905664243] exists,up ed061345-8a22-457d-a3dc-13d2ee757388 2024-04-22T01:12:51.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6816/3257841658,v1:172.21.15.111:6817/3257841658] [v2:172.21.15.111:6818/3257841658,v1:172.21.15.111:6819/3257841658] exists,up c9832e0c-b8f2-42d4-a384-a43975ec432a 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/3416662120,v1:172.21.15.203:6817/3416662120] [v2:172.21.15.203:6818/3416662120,v1:172.21.15.203:6819/3416662120] exists,up b4a5470d-0f1c-45dc-a692-cd4144caf225 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6826/1134107438,v1:172.21.15.38:6827/1134107438] [v2:172.21.15.38:6828/1134107438,v1:172.21.15.38:6829/1134107438] exists,up d125ca4b-b70c-471e-864b-6eb587daf211 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 up in weight 1 up_from 72 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6824/1666422758,v1:172.21.15.111:6825/1666422758] [v2:172.21.15.111:6826/1666422758,v1:172.21.15.111:6827/1666422758] exists,up 23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 up in weight 1 up_from 80 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/3028852756,v1:172.21.15.203:6825/3028852756] [v2:172.21.15.203:6826/3028852756,v1:172.21.15.203:6827/3028852756] exists,up 60ce13a3-6d11-48e8-b53e-609c433deaf3 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.4 [2,11] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.6 [8,11] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.16 [2,4] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.1f [8,1] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.0 [4,8] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.6 [4,6] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.c [3,9] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.10 [5,9] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.14 [4,7] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.2a [3,8] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.33 [5,9] 2024-04-22T01:12:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.f [9,10,5,8] 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.13 [9,8] 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.19 [9,8] 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/3063830265 expires 2024-04-23T01:03:15.695973+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/3063830265 expires 2024-04-23T01:03:15.695973+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2675569647 expires 2024-04-23T01:03:15.695973+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2608977615 expires 2024-04-23T01:03:15.695973+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/3674504753 expires 2024-04-23T01:02:47.985940+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/3674504753 expires 2024-04-23T01:02:47.985940+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/799365618 expires 2024-04-23T01:02:47.985940+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3519840670 expires 2024-04-23T01:02:47.985940+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/845318699 expires 2024-04-23T01:02:47.985940+0000 2024-04-22T01:12:51.721 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/687303682 expires 2024-04-23T01:03:15.695973+0000 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.719+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe3c0a2c20 msgr2=0x7efe3c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.719+0000 7efe5944b700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe3c0a2c20 0x7efe3c0a50e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe440104e0 tx=0x7efe4400b3c0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.719+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 msgr2=0x7efe541a1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.719+0000 7efe5944b700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe541a1bd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00d010 tx=0x7efe4c004dd0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 shutdown_connections 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efe3c0a2c20 0x7efe3c0a50e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe54101eb0 0x7efe541a1bd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe54100b90 0x7efe5419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 --2- 172.21.15.38:0/2060800960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe541014f0 0x7efe5419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:51.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 >> 172.21.15.38:0/2060800960 conn(0x7efe540fc750 msgr2=0x7efe540fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:51.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 shutdown_connections 2024-04-22T01:12:51.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:51.720+0000 7efe5944b700 1 -- 172.21.15.38:0/2060800960 wait complete. 2024-04-22T01:12:51.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:51 smithi038 ceph-mon[28043]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-22T01:12:52.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:51 smithi203 ceph-mon[31825]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-22T01:12:52.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:51 smithi111 ceph-mon[31057]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-04-22T01:12:52.394 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-22T01:12:52.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.514+0000 7faa5925f640 1 -- 172.21.15.38:0/29518319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa54102da0 msgr2=0x7faa541031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.514+0000 7faa5925f640 1 --2- 172.21.15.38:0/29518319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa54102da0 0x7faa541031c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faa44009ca0 tx=0x7faa4402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.514+0000 7faa5925f640 1 -- 172.21.15.38:0/29518319 shutdown_connections 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.514+0000 7faa5925f640 1 --2- 172.21.15.38:0/29518319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa541001d0 0x7faa54109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.514+0000 7faa5925f640 1 --2- 172.21.15.38:0/29518319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa540ff810 0x7faa540ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.514+0000 7faa5925f640 1 --2- 172.21.15.38:0/29518319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa54102da0 0x7faa541031c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.515+0000 7faa5925f640 1 -- 172.21.15.38:0/29518319 >> 172.21.15.38:0/29518319 conn(0x7faa540fb3b0 msgr2=0x7faa540fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.515+0000 7faa5925f640 1 -- 172.21.15.38:0/29518319 shutdown_connections 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.515+0000 7faa5925f640 1 -- 172.21.15.38:0/29518319 wait complete. 2024-04-22T01:12:52.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.515+0000 7faa5925f640 1 Processor -- start 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.515+0000 7faa5925f640 1 -- start start 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 0x7faa54197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa541001d0 0x7faa54198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa54102da0 0x7faa5419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa5410d4a0 con 0x7faa540ff810 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa5410d320 con 0x7faa541001d0 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa5410d620 con 0x7faa54102da0 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 0x7faa54197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 0x7faa54197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41008/0 (socket says 172.21.15.38:41008) 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 -- 172.21.15.38:0/1013131265 learned_addr learned my addr 172.21.15.38:0/1013131265 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5259c640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa541001d0 0x7faa54198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 -- 172.21.15.38:0/1013131265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa54102da0 msgr2=0x7faa5419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa54102da0 0x7faa5419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 -- 172.21.15.38:0/1013131265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa541001d0 msgr2=0x7faa54198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa541001d0 0x7faa54198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 -- 172.21.15.38:0/1013131265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5419f730 con 0x7faa540ff810 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa52d9d640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 0x7faa54197b10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faa440324c0 tx=0x7faa44032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44039040 con 0x7faa540ff810 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa44044070 con 0x7faa540ff810 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5419f9c0 con 0x7faa540ff810 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa440042a0 con 0x7faa540ff810 2024-04-22T01:12:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.516+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5419feb0 con 0x7faa540ff810 2024-04-22T01:12:52.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.517+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa54108ce0 con 0x7faa540ff810 2024-04-22T01:12:52.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.519+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faa44003970 con 0x7faa540ff810 2024-04-22T01:12:52.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.519+0000 7faa3bfff640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa3c0a2b90 0x7faa3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.519+0000 7faa5259c640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa3c0a2b90 0x7faa3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:52.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.519+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7faa440f18f0 con 0x7faa540ff810 2024-04-22T01:12:52.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.520+0000 7faa5259c640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa3c0a2b90 0x7faa3c0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa54199020 tx=0x7faa4800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:52.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.524+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7faa440be510 con 0x7faa540ff810 2024-04-22T01:12:52.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.715+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa54062340 con 0x7faa540ff810 2024-04-22T01:12:52.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.716+0000 7faa3bfff640 1 -- 172.21.15.38:0/1013131265 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7faa440bde10 con 0x7faa540ff810 2024-04-22T01:12:52.717 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:52.717 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa3c0a2b90 msgr2=0x7faa3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa3c0a2b90 0x7faa3c0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa54199020 tx=0x7faa4800b390 comp rx=0 tx=0).stop 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 msgr2=0x7faa54197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 0x7faa54197b10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faa440324c0 tx=0x7faa44032f20 comp rx=0 tx=0).stop 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 shutdown_connections 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa3c0a2b90 0x7faa3c0a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa54102da0 0x7faa5419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa541001d0 0x7faa54198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 --2- 172.21.15.38:0/1013131265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa540ff810 0x7faa54197b10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 >> 172.21.15.38:0/1013131265 conn(0x7faa540fb3b0 msgr2=0x7faa54105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:52.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 shutdown_connections 2024-04-22T01:12:52.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.719+0000 7faa5925f640 1 -- 172.21.15.38:0/1013131265 wait complete. 2024-04-22T01:12:52.731 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:12:52.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.836+0000 7f8fdec39640 1 -- 172.21.15.38:0/3361847127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8107460 msgr2=0x7f8fd81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.836+0000 7f8fdec39640 1 --2- 172.21.15.38:0/3361847127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8107460 0x7f8fd81098b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009ca0 tx=0x7f8fd002c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- 172.21.15.38:0/3361847127 shutdown_connections 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 --2- 172.21.15.38:0/3361847127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8107460 0x7f8fd81098b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 --2- 172.21.15.38:0/3361847127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 0x7f8fd8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 --2- 172.21.15.38:0/3361847127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd8102200 0x7f8fd81045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- 172.21.15.38:0/3361847127 >> 172.21.15.38:0/3361847127 conn(0x7f8fd80fbbf0 msgr2=0x7f8fd80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- 172.21.15.38:0/3361847127 shutdown_connections 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- 172.21.15.38:0/3361847127 wait complete. 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 Processor -- start 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- start start 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 0x7f8fd8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 0x7f8fd8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd8107460 0x7f8fd819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd810d420 con 0x7f8fd8102200 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd810d2a0 con 0x7f8fd8107460 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdec39640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd810d5a0 con 0x7f8fd8104b30 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdc9d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 0x7f8fd8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdc9d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 0x7f8fd8197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41034/0 (socket says 172.21.15.38:41034) 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.837+0000 7f8fdc9d5640 1 -- 172.21.15.38:0/1025630966 learned_addr learned my addr 172.21.15.38:0/1025630966 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fd7fff640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 0x7f8fd8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdc9d5640 1 -- 172.21.15.38:0/1025630966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 msgr2=0x7f8fd8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdc9d5640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 0x7f8fd8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdc9d5640 1 -- 172.21.15.38:0/1025630966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd8107460 msgr2=0x7f8fd819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdc9d5640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd8107460 0x7f8fd819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdc9d5640 1 -- 172.21.15.38:0/1025630966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd819f710 con 0x7f8fd8102200 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fd7fff640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 0x7f8fd8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdc9d5640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 0x7f8fd8197af0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800d950 tx=0x7f8fc800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8017030 con 0x7f8fd8102200 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc8004020 con 0x7f8fd8102200 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd819fa00 con 0x7f8fd8102200 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80047a0 con 0x7f8fd8102200 2024-04-22T01:12:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.838+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd819ff50 con 0x7f8fd8102200 2024-04-22T01:12:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.839+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd80fd6e0 con 0x7f8fd8102200 2024-04-22T01:12:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.840+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8fc80041d0 con 0x7f8fd8102200 2024-04-22T01:12:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.841+0000 7f8fd5ffb640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fc00a2b90 0x7f8fc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.841+0000 7f8fd7fff640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fc00a2b90 0x7f8fc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.841+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f8fc80cef90 con 0x7f8fd8102200 2024-04-22T01:12:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.842+0000 7f8fd7fff640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fc00a2b90 0x7f8fc00a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc005fd0 tx=0x7f8fcc0094d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:52.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:52.846+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8fc809bae0 con 0x7f8fd8102200 2024-04-22T01:12:53.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:52 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2060800960' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T01:12:53.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.031+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8fd8062340 con 0x7f8fd8102200 2024-04-22T01:12:53.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.031+0000 7f8fd5ffb640 1 -- 172.21.15.38:0/1025630966 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f8fc809b3e0 con 0x7f8fd8102200 2024-04-22T01:12:53.032 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fc00a2b90 msgr2=0x7f8fc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fc00a2b90 0x7f8fc00a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc005fd0 tx=0x7f8fcc0094d0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 msgr2=0x7f8fd8197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 0x7f8fd8197af0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800d950 tx=0x7f8fc800de10 comp rx=0 tx=0).stop 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 shutdown_connections 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8fc00a2b90 0x7f8fc00a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd8107460 0x7f8fd819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd8104b30 0x7f8fd8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 --2- 172.21.15.38:0/1025630966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8fd8102200 0x7f8fd8197af0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 >> 172.21.15.38:0/1025630966 conn(0x7f8fd80fbbf0 msgr2=0x7f8fd8102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 shutdown_connections 2024-04-22T01:12:53.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.034+0000 7f8fdec39640 1 -- 172.21.15.38:0/1025630966 wait complete. 2024-04-22T01:12:53.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:52 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2060800960' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T01:12:53.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:12:53.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:52 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2060800960' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T01:12:53.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 -- 172.21.15.38:0/2204420866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 msgr2=0x7f448010baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 --2- 172.21.15.38:0/2204420866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f448010baf0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4478009ca0 tx=0x7f447802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 -- 172.21.15.38:0/2204420866 shutdown_connections 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 --2- 172.21.15.38:0/2204420866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f448010baf0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 --2- 172.21.15.38:0/2204420866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44800741a0 0x7f4480074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 --2- 172.21.15.38:0/2204420866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4480075780 0x7f4480073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 -- 172.21.15.38:0/2204420866 >> 172.21.15.38:0/2204420866 conn(0x7f44800fcd10 msgr2=0x7f44800ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 -- 172.21.15.38:0/2204420866 shutdown_connections 2024-04-22T01:12:53.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.149+0000 7f4484a96640 1 -- 172.21.15.38:0/2204420866 wait complete. 2024-04-22T01:12:53.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 Processor -- start 2024-04-22T01:12:53.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 -- start start 2024-04-22T01:12:53.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44800741a0 0x7f4480199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4480075780 0x7f448019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f44801a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f448010f660 con 0x7f4480109660 2024-04-22T01:12:53.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f448010f4e0 con 0x7f4480075780 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4484a96640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f448010f7e0 con 0x7f44800741a0 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f447ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f44801a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f447ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f44801a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41048/0 (socket says 172.21.15.38:41048) 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f447ed9d640 1 -- 172.21.15.38:0/2949148607 learned_addr learned my addr 172.21.15.38:0/2949148607 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f447e59c640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44800741a0 0x7f4480199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.150+0000 7f4475d9b640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4480075780 0x7f448019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f447ed9d640 1 -- 172.21.15.38:0/2949148607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44800741a0 msgr2=0x7f4480199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f447ed9d640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44800741a0 0x7f4480199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f447ed9d640 1 -- 172.21.15.38:0/2949148607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4480075780 msgr2=0x7f448019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f447ed9d640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4480075780 0x7f448019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f447ed9d640 1 -- 172.21.15.38:0/2949148607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44801a18c0 con 0x7f4480109660 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f447ed9d640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f44801a1220 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4478037040 tx=0x7f4478032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478039040 con 0x7f4480109660 2024-04-22T01:12:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4478044070 con 0x7f4480109660 2024-04-22T01:12:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44801a1b50 con 0x7f4480109660 2024-04-22T01:12:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478004260 con 0x7f4480109660 2024-04-22T01:12:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.151+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44801a2040 con 0x7f4480109660 2024-04-22T01:12:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.152+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4480108390 con 0x7f4480109660 2024-04-22T01:12:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.153+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4478003970 con 0x7f4480109660 2024-04-22T01:12:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.154+0000 7f4477fff640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f446c0a2b90 0x7f446c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.154+0000 7f447e59c640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f446c0a2b90 0x7f446c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.154+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f44780f1b00 con 0x7f4480109660 2024-04-22T01:12:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.154+0000 7f447e59c640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f446c0a2b90 0x7f446c0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f44700058c0 tx=0x7f447000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.158+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f44780be690 con 0x7f4480109660 2024-04-22T01:12:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.341+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4480062340 con 0x7f4480109660 2024-04-22T01:12:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.341+0000 7f4477fff640 1 -- 172.21.15.38:0/2949148607 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f44780bdf90 con 0x7f4480109660 2024-04-22T01:12:53.342 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f446c0a2b90 msgr2=0x7f446c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f446c0a2b90 0x7f446c0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f44700058c0 tx=0x7f447000a2d0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 msgr2=0x7f44801a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f44801a1220 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4478037040 tx=0x7f4478032f90 comp rx=0 tx=0).stop 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 shutdown_connections 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f446c0a2b90 0x7f446c0a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4480109660 0x7f44801a1220 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4480075780 0x7f448019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 --2- 172.21.15.38:0/2949148607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44800741a0 0x7f4480199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.343+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 >> 172.21.15.38:0/2949148607 conn(0x7f44800fcd10 msgr2=0x7f4480105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.344+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 shutdown_connections 2024-04-22T01:12:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.344+0000 7f4484a96640 1 -- 172.21.15.38:0/2949148607 wait complete. 2024-04-22T01:12:53.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:12:53.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 -- 172.21.15.38:0/2332678400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c102250 msgr2=0x7fd04c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 --2- 172.21.15.38:0/2332678400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c102250 0x7fd04c108ae0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd044009ca0 tx=0x7fd04402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 -- 172.21.15.38:0/2332678400 shutdown_connections 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 --2- 172.21.15.38:0/2332678400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c102250 0x7fd04c108ae0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 --2- 172.21.15.38:0/2332678400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c101890 0x7fd04c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 --2- 172.21.15.38:0/2332678400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c1006a0 0x7fd04c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 -- 172.21.15.38:0/2332678400 >> 172.21.15.38:0/2332678400 conn(0x7fd04c0fbc30 msgr2=0x7fd04c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 -- 172.21.15.38:0/2332678400 shutdown_connections 2024-04-22T01:12:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.460+0000 7fd050f1e640 1 -- 172.21.15.38:0/2332678400 wait complete. 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 Processor -- start 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 -- start start 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 0x7fd04c196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c101890 0x7fd04c196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c102250 0x7fd04c19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c10c300 con 0x7fd04c1006a0 2024-04-22T01:12:53.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c10c180 con 0x7fd04c102250 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd050f1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c10c480 con 0x7fd04c101890 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 0x7fd04c196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 0x7fd04c196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41064/0 (socket says 172.21.15.38:41064) 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 -- 172.21.15.38:0/1594675477 learned_addr learned my addr 172.21.15.38:0/1594675477 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd049d9b640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c101890 0x7fd04c196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 -- 172.21.15.38:0/1594675477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c101890 msgr2=0x7fd04c196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c101890 0x7fd04c196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04ad9d640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c102250 0x7fd04c19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 -- 172.21.15.38:0/1594675477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c102250 msgr2=0x7fd04c19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c102250 0x7fd04c19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd04a59c640 1 -- 172.21.15.38:0/1594675477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04c19e450 con 0x7fd04c1006a0 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.461+0000 7fd049d9b640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c101890 0x7fd04c196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:53.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.462+0000 7fd04a59c640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 0x7fd04c196830 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c970 tx=0x7fd03c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:53.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.462+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c007c10 con 0x7fd04c1006a0 2024-04-22T01:12:53.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.462+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd03c007db0 con 0x7fd04c1006a0 2024-04-22T01:12:53.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.462+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04c19e740 con 0x7fd04c1006a0 2024-04-22T01:12:53.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.462+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c00f940 con 0x7fd04c1006a0 2024-04-22T01:12:53.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.462+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd04c19ec90 con 0x7fd04c1006a0 2024-04-22T01:12:53.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.463+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd04c1078f0 con 0x7fd04c1006a0 2024-04-22T01:12:53.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.464+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd03c00fae0 con 0x7fd04c1006a0 2024-04-22T01:12:53.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.464+0000 7fd03b7fe640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0340a2b90 0x7fd0340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.465+0000 7fd049d9b640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0340a2b90 0x7fd0340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.465+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd03c0cf840 con 0x7fd04c1006a0 2024-04-22T01:12:53.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.465+0000 7fd049d9b640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0340a2b90 0x7fd0340a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd04c197d40 tx=0x7fd04000a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:53.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.469+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd03c09c460 con 0x7fd04c1006a0 2024-04-22T01:12:53.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.655+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd04c062370 con 0x7fd04c1006a0 2024-04-22T01:12:53.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.656+0000 7fd03b7fe640 1 -- 172.21.15.38:0/1594675477 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fd03c09bd60 con 0x7fd04c1006a0 2024-04-22T01:12:53.656 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:12:53.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.657+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0340a2b90 msgr2=0x7fd0340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.657+0000 7fd050f1e640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0340a2b90 0x7fd0340a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd04c197d40 tx=0x7fd04000a5c0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.657+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 msgr2=0x7fd04c196830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.657+0000 7fd050f1e640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 0x7fd04c196830 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c970 tx=0x7fd03c00ce30 comp rx=0 tx=0).stop 2024-04-22T01:12:53.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 shutdown_connections 2024-04-22T01:12:53.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd0340a2b90 0x7fd0340a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04c102250 0x7fd04c19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd04c101890 0x7fd04c196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 --2- 172.21.15.38:0/1594675477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd04c1006a0 0x7fd04c196830 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 >> 172.21.15.38:0/1594675477 conn(0x7fd04c0fbc30 msgr2=0x7fd04c104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 shutdown_connections 2024-04-22T01:12:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.658+0000 7fd050f1e640 1 -- 172.21.15.38:0/1594675477 wait complete. 2024-04-22T01:12:53.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:12:53.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1808485111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 msgr2=0x7fa774100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1808485111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774100ac0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa764009900 tx=0x7fa76402bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:53.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1808485111 shutdown_connections 2024-04-22T01:12:53.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1808485111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa774102250 0x7fa774108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1808485111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa774101890 0x7fa774101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1808485111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774100ac0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1808485111 >> 172.21.15.38:0/1808485111 conn(0x7fa7740fbc30 msgr2=0x7fa7740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1808485111 shutdown_connections 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1808485111 wait complete. 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 Processor -- start 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.776+0000 7fa77a8b5640 1 -- start start 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa77a8b5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa77a8b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa774101890 0x7fa774198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa77a8b5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa774102250 0x7fa77419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa77a8b5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa77410c370 con 0x7fa7741006a0 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa77a8b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa77410c1f0 con 0x7fa774101890 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa77a8b5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa77410c4f0 con 0x7fa774102250 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41084/0 (socket says 172.21.15.38:41084) 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 -- 172.21.15.38:0/1536008011 learned_addr learned my addr 172.21.15.38:0/1536008011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa7737fe640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa774101890 0x7fa774198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa778e52640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa774102250 0x7fa77419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 -- 172.21.15.38:0/1536008011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa774102250 msgr2=0x7fa77419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa774102250 0x7fa77419f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 -- 172.21.15.38:0/1536008011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa774101890 msgr2=0x7fa774198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa774101890 0x7fa774198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 -- 172.21.15.38:0/1536008011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77419f7c0 con 0x7fa7741006a0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.777+0000 7fa773fff640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774197bf0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa764005ab0 tx=0x7fa764005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.778+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764041070 con 0x7fa7741006a0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.778+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7640054e0 con 0x7fa7741006a0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.778+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77419fa50 con 0x7fa7741006a0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.778+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764034440 con 0x7fa7741006a0 2024-04-22T01:12:53.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.778+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa77419ff40 con 0x7fa7741006a0 2024-04-22T01:12:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.779+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7741a0110 con 0x7fa7741006a0 2024-04-22T01:12:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.780+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa764044070 con 0x7fa7741006a0 2024-04-22T01:12:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.780+0000 7fa7717fa640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa75c0a2b90 0x7fa75c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.780+0000 7fa7737fe640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa75c0a2b90 0x7fa75c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.780+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa7640f10f0 con 0x7fa7741006a0 2024-04-22T01:12:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.781+0000 7fa7737fe640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa75c0a2b90 0x7fa75c0a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7741990b0 tx=0x7fa768009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:53.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.785+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa7640bdc80 con 0x7fa7741006a0 2024-04-22T01:12:53.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.969+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa7741a04a0 con 0x7fa7741006a0 2024-04-22T01:12:53.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.969+0000 7fa7717fa640 1 -- 172.21.15.38:0/1536008011 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fa7640bd580 con 0x7fa7741006a0 2024-04-22T01:12:53.969 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa75c0a2b90 msgr2=0x7fa75c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa75c0a2b90 0x7fa75c0a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7741990b0 tx=0x7fa768009110 comp rx=0 tx=0).stop 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 msgr2=0x7fa774197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774197bf0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa764005ab0 tx=0x7fa764005dc0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 shutdown_connections 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa75c0a2b90 0x7fa75c0a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa774102250 0x7fa77419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa774101890 0x7fa774198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 --2- 172.21.15.38:0/1536008011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7741006a0 0x7fa774197bf0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 >> 172.21.15.38:0/1536008011 conn(0x7fa7740fbc30 msgr2=0x7fa7740fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 shutdown_connections 2024-04-22T01:12:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:53.971+0000 7fa77a8b5640 1 -- 172.21.15.38:0/1536008011 wait complete. 2024-04-22T01:12:53.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:53 smithi038 ceph-mon[28043]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T01:12:53.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1013131265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:53.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1025630966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:53.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2949148607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:53.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1594675477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:53.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:12:54.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:53 smithi203 ceph-mon[31825]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T01:12:54.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1013131265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:54.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1025630966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2949148607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1594675477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 -- 172.21.15.38:0/1075522179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 msgr2=0x7f4148106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 --2- 172.21.15.38:0/1075522179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148106f40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4138009900 tx=0x7f413802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 -- 172.21.15.38:0/1075522179 shutdown_connections 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 --2- 172.21.15.38:0/1075522179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4148107480 0x7f41481098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 --2- 172.21.15.38:0/1075522179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148106f40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 --2- 172.21.15.38:0/1075522179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4148102220 0x7f4148104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.084+0000 7f414ff97640 1 -- 172.21.15.38:0/1075522179 >> 172.21.15.38:0/1075522179 conn(0x7f41480fbc30 msgr2=0x7f41480fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 -- 172.21.15.38:0/1075522179 shutdown_connections 2024-04-22T01:12:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 -- 172.21.15.38:0/1075522179 wait complete. 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 Processor -- start 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 -- start start 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4148102220 0x7f4148197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4148107480 0x7f414819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f414810d410 con 0x7f4148104b50 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f414810d290 con 0x7f4148107480 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414ff97640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f414810d590 con 0x7f4148102220 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414d532640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414d532640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41098/0 (socket says 172.21.15.38:41098) 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414d532640 1 -- 172.21.15.38:0/552159637 learned_addr learned my addr 172.21.15.38:0/552159637 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.085+0000 7f414e534640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4148107480 0x7f414819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414dd33640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4148102220 0x7f4148197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414d532640 1 -- 172.21.15.38:0/552159637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4148102220 msgr2=0x7f4148197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414d532640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4148102220 0x7f4148197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414d532640 1 -- 172.21.15.38:0/552159637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4148107480 msgr2=0x7f414819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414d532640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4148107480 0x7f414819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414d532640 1 -- 172.21.15.38:0/552159637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414819f680 con 0x7f4148104b50 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414d532640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148197fa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4138000c00 tx=0x7f4138005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138041070 con 0x7f4148104b50 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41380044d0 con 0x7f4148104b50 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414819f910 con 0x7f4148104b50 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138034440 con 0x7f4148104b50 2024-04-22T01:12:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.086+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414819fe50 con 0x7f4148104b50 2024-04-22T01:12:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.087+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41480fd720 con 0x7f4148104b50 2024-04-22T01:12:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.088+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4138044070 con 0x7f4148104b50 2024-04-22T01:12:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.089+0000 7f413effd640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f41340a2c10 0x7f41340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.089+0000 7f414dd33640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f41340a2c10 0x7f41340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.089+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f41380f0f30 con 0x7f4148104b50 2024-04-22T01:12:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.089+0000 7f414dd33640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f41340a2c10 0x7f41340a50d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4144009b60 tx=0x7f4144005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:54.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.093+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f41380bdac0 con 0x7f4148104b50 2024-04-22T01:12:54.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:53 smithi111 ceph-mon[31057]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T01:12:54.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1013131265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:54.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1025630966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2949148607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1594675477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.317+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f414802d170 con 0x7f4148104b50 2024-04-22T01:12:54.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.318+0000 7f413effd640 1 -- 172.21.15.38:0/552159637 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f41380bd3c0 con 0x7f4148104b50 2024-04-22T01:12:54.318 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:54.318 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:11:00.429607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24451,24472,24446,24527,24485]},"id":1}]} 2024-04-22T01:12:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:12:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f41340a2c10 msgr2=0x7f41340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f41340a2c10 0x7f41340a50d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4144009b60 tx=0x7f4144005f90 comp rx=0 tx=0).stop 2024-04-22T01:12:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 msgr2=0x7f4148197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148197fa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4138000c00 tx=0x7f4138005d50 comp rx=0 tx=0).stop 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 shutdown_connections 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f41340a2c10 0x7f41340a50d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4148107480 0x7f414819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4148104b50 0x7f4148197fa0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 --2- 172.21.15.38:0/552159637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4148102220 0x7f4148197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 >> 172.21.15.38:0/552159637 conn(0x7f41480fbc30 msgr2=0x7f4148102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 shutdown_connections 2024-04-22T01:12:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.320+0000 7f414ff97640 1 -- 172.21.15.38:0/552159637 wait complete. 2024-04-22T01:12:54.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:12:54.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.433+0000 7f79cc439640 1 -- 172.21.15.38:0/1199917994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4104b30 msgr2=0x7f79c4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.433+0000 7f79cc439640 1 --2- 172.21.15.38:0/1199917994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4104b30 0x7f79c4106f20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009900 tx=0x7f79b402bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:54.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- 172.21.15.38:0/1199917994 shutdown_connections 2024-04-22T01:12:54.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 --2- 172.21.15.38:0/1199917994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 0x7f79c41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 --2- 172.21.15.38:0/1199917994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4104b30 0x7f79c4106f20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 --2- 172.21.15.38:0/1199917994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79c4102200 0x7f79c41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- 172.21.15.38:0/1199917994 >> 172.21.15.38:0/1199917994 conn(0x7f79c40fbbf0 msgr2=0x7f79c40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- 172.21.15.38:0/1199917994 shutdown_connections 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- 172.21.15.38:0/1199917994 wait complete. 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 Processor -- start 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- start start 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 0x7f79c4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79c4104b30 0x7f79c4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 0x7f79c419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f79c410d450 con 0x7f79c4102200 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f79c410d2d0 con 0x7f79c4104b30 2024-04-22T01:12:54.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79cc439640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79c410d5d0 con 0x7f79c4107460 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79ca1d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 0x7f79c4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79ca1d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 0x7f79c4197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41114/0 (socket says 172.21.15.38:41114) 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.434+0000 7f79ca1d5640 1 -- 172.21.15.38:0/1424941411 learned_addr learned my addr 172.21.15.38:0/1424941411 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79c99d4640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79c4104b30 0x7f79c4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca9d6640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 0x7f79c419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca1d5640 1 -- 172.21.15.38:0/1424941411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 msgr2=0x7f79c419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca1d5640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 0x7f79c419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca1d5640 1 -- 172.21.15.38:0/1424941411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79c4104b30 msgr2=0x7f79c4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca1d5640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79c4104b30 0x7f79c4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca1d5640 1 -- 172.21.15.38:0/1424941411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c419f6f0 con 0x7f79c4102200 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca9d6640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 0x7f79c419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79ca1d5640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 0x7f79c4197ad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f79c000d950 tx=0x7f79c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0017070 con 0x7f79c4102200 2024-04-22T01:12:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c419f9e0 con 0x7f79c4102200 2024-04-22T01:12:54.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c0009c30 con 0x7f79c4102200 2024-04-22T01:12:54.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c000f6b0 con 0x7f79c4102200 2024-04-22T01:12:54.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.435+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c419ff00 con 0x7f79c4102200 2024-04-22T01:12:54.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.436+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c40fd6e0 con 0x7f79c4102200 2024-04-22T01:12:54.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.437+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f79c00109d0 con 0x7f79c4102200 2024-04-22T01:12:54.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.438+0000 7f79bb7fe640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f79b00a2b90 0x7f79b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.438+0000 7f79c99d4640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f79b00a2b90 0x7f79b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.438+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f79c00cf130 con 0x7f79c4102200 2024-04-22T01:12:54.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.438+0000 7f79c99d4640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f79b00a2b90 0x7f79b00a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f79b402c170 tx=0x7f79b402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:54.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.442+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f79c0009de0 con 0x7f79c4102200 2024-04-22T01:12:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.630+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79c4062340 con 0x7f79c4102200 2024-04-22T01:12:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.632+0000 7f79bb7fe640 1 -- 172.21.15.38:0/1424941411 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7f79c009bcc0 con 0x7f79c4102200 2024-04-22T01:12:54.632 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:54.632 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:12:54.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f79b00a2b90 msgr2=0x7f79b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f79b00a2b90 0x7f79b00a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f79b402c170 tx=0x7f79b402df90 comp rx=0 tx=0).stop 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 msgr2=0x7f79c4197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 0x7f79c4197ad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f79c000d950 tx=0x7f79c000de10 comp rx=0 tx=0).stop 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 shutdown_connections 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f79b00a2b90 0x7f79b00a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79c4107460 0x7f79c419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79c4104b30 0x7f79c4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 --2- 172.21.15.38:0/1424941411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79c4102200 0x7f79c4197ad0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 >> 172.21.15.38:0/1424941411 conn(0x7f79c40fbbf0 msgr2=0x7f79c4102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 shutdown_connections 2024-04-22T01:12:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.634+0000 7f79cc439640 1 -- 172.21.15.38:0/1424941411 wait complete. 2024-04-22T01:12:54.646 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:12:54.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:12:54.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 -- 172.21.15.38:0/2265242397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8102230 msgr2=0x7f86f8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2265242397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8102230 0x7f86f8108ac0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009900 tx=0x7f86f002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 -- 172.21.15.38:0/2265242397 shutdown_connections 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2265242397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8102230 0x7f86f8108ac0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2265242397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8101870 0x7f86f8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2265242397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8100680 0x7f86f8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 -- 172.21.15.38:0/2265242397 >> 172.21.15.38:0/2265242397 conn(0x7f86f80fbbf0 msgr2=0x7f86f80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 -- 172.21.15.38:0/2265242397 shutdown_connections 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.752+0000 7f86ffecf640 1 -- 172.21.15.38:0/2265242397 wait complete. 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 Processor -- start 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 -- start start 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 0x7f86f8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8101870 0x7f86f8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8102230 0x7f86f819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f86f810d4c0 con 0x7f86f8100680 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f86f810d340 con 0x7f86f8102230 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86ffecf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86f810d640 con 0x7f86f8101870 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 0x7f86f8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 0x7f86f8197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41134/0 (socket says 172.21.15.38:41134) 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 -- 172.21.15.38:0/2079233188 learned_addr learned my addr 172.21.15.38:0/2079233188 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fd46a640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8101870 0x7f86f8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fe46c640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8102230 0x7f86f819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 -- 172.21.15.38:0/2079233188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8101870 msgr2=0x7f86f8197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8101870 0x7f86f8197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 -- 172.21.15.38:0/2079233188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8102230 msgr2=0x7f86f819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8102230 0x7f86f819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 -- 172.21.15.38:0/2079233188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f819f690 con 0x7f86f8100680 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fe46c640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8102230 0x7f86f819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fd46a640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8101870 0x7f86f8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:54.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.753+0000 7f86fdc6b640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 0x7f86f8197a70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00b9c0 tx=0x7f86ec00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.754+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec019070 con 0x7f86f8100680 2024-04-22T01:12:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.754+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f819f980 con 0x7f86f8100680 2024-04-22T01:12:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.754+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ec00cba0 con 0x7f86f8100680 2024-04-22T01:12:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.754+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec012620 con 0x7f86f8100680 2024-04-22T01:12:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.754+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f819fed0 con 0x7f86f8100680 2024-04-22T01:12:54.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.755+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f8107940 con 0x7f86f8100680 2024-04-22T01:12:54.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.756+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f86ec01c070 con 0x7f86f8100680 2024-04-22T01:12:54.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.757+0000 7f86f6ffd640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86e00a2b90 0x7f86e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:54.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.757+0000 7f86fd46a640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86e00a2b90 0x7f86e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:54.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.757+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f86ec0cf060 con 0x7f86f8100680 2024-04-22T01:12:54.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.757+0000 7f86fd46a640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86e00a2b90 0x7f86e00a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f86f8198f80 tx=0x7f86e800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:54.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.761+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f86ec09bbf0 con 0x7f86f8100680 2024-04-22T01:12:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1536008011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/552159637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:54.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:54 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1424941411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:54.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.944+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86f80623e0 con 0x7f86f8100680 2024-04-22T01:12:54.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.944+0000 7f86f6ffd640 1 -- 172.21.15.38:0/2079233188 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f86ec09b4f0 con 0x7f86f8100680 2024-04-22T01:12:54.944 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:12:54.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86e00a2b90 msgr2=0x7f86e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86e00a2b90 0x7f86e00a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f86f8198f80 tx=0x7f86e800a5c0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 msgr2=0x7f86f8197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 0x7f86f8197a70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00b9c0 tx=0x7f86ec00be80 comp rx=0 tx=0).stop 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 shutdown_connections 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86e00a2b90 0x7f86e00a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86f8102230 0x7f86f819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86f8101870 0x7f86f8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 --2- 172.21.15.38:0/2079233188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86f8100680 0x7f86f8197a70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 >> 172.21.15.38:0/2079233188 conn(0x7f86f80fbbf0 msgr2=0x7f86f8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 shutdown_connections 2024-04-22T01:12:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:54.946+0000 7f86ffecf640 1 -- 172.21.15.38:0/2079233188 wait complete. 2024-04-22T01:12:54.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:12:55.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1536008011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:55.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/552159637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:55.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:54 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1424941411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:55.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 -- 172.21.15.38:0/747070430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20103350 msgr2=0x7fae20109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 --2- 172.21.15.38:0/747070430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20103350 0x7fae20109be0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fae18009ca0 tx=0x7fae1802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 -- 172.21.15.38:0/747070430 shutdown_connections 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 --2- 172.21.15.38:0/747070430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20103350 0x7fae20109be0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 --2- 172.21.15.38:0/747070430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae20102990 0x7fae20102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 --2- 172.21.15.38:0/747070430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae201017a0 0x7fae20101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 -- 172.21.15.38:0/747070430 >> 172.21.15.38:0/747070430 conn(0x7fae200fcd10 msgr2=0x7fae200ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 -- 172.21.15.38:0/747070430 shutdown_connections 2024-04-22T01:12:55.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.063+0000 7fae24d25640 1 -- 172.21.15.38:0/747070430 wait complete. 2024-04-22T01:12:55.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 Processor -- start 2024-04-22T01:12:55.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 -- start start 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae201017a0 0x7fae200754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 0x7fae20073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae20103350 0x7fae20074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d510 con 0x7fae20102990 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d390 con 0x7fae20103350 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae24d25640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d690 con 0x7fae201017a0 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 0x7fae20073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 0x7fae20073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41144/0 (socket says 172.21.15.38:41144) 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 -- 172.21.15.38:0/3593373139 learned_addr learned my addr 172.21.15.38:0/3593373139 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1e59c640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae201017a0 0x7fae200754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 -- 172.21.15.38:0/3593373139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae201017a0 msgr2=0x7fae200754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae201017a0 0x7fae200754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 -- 172.21.15.38:0/3593373139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae20103350 msgr2=0x7fae20074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae20103350 0x7fae20074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.064+0000 7fae1dd9b640 1 -- 172.21.15.38:0/3593373139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae20074900 con 0x7fae20102990 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.065+0000 7fae1dd9b640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 0x7fae20073b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b4a0 tx=0x7fae1400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.065+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14014070 con 0x7fae20102990 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.065+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae14010af0 con 0x7fae20102990 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.065+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae201a1c40 con 0x7fae20102990 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.065+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14017d10 con 0x7fae20102990 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.065+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae201a2210 con 0x7fae20102990 2024-04-22T01:12:55.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.066+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae20108ac0 con 0x7fae20102990 2024-04-22T01:12:55.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.067+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fae14017070 con 0x7fae20102990 2024-04-22T01:12:55.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.067+0000 7fae0f7fe640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fae080a2c10 0x7fae080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.068+0000 7fae1e59c640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fae080a2c10 0x7fae080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.068+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fae140ce7f0 con 0x7fae20102990 2024-04-22T01:12:55.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.068+0000 7fae1e59c640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fae080a2c10 0x7fae080a50d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fae201027f0 tx=0x7fae10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.072+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fae1409b380 con 0x7fae20102990 2024-04-22T01:12:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1536008011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/552159637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:54 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1424941411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:55.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.256+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fae20062370 con 0x7fae20102990 2024-04-22T01:12:55.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.257+0000 7fae0f7fe640 1 -- 172.21.15.38:0/3593373139 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fae1409ac80 con 0x7fae20102990 2024-04-22T01:12:55.257 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.258+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fae080a2c10 msgr2=0x7fae080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.258+0000 7fae24d25640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fae080a2c10 0x7fae080a50d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fae201027f0 tx=0x7fae10008040 comp rx=0 tx=0).stop 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.258+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 msgr2=0x7fae20073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.258+0000 7fae24d25640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 0x7fae20073b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b4a0 tx=0x7fae1400b960 comp rx=0 tx=0).stop 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 shutdown_connections 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fae080a2c10 0x7fae080a50d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae20103350 0x7fae20074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae20102990 0x7fae20073b30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 --2- 172.21.15.38:0/3593373139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae201017a0 0x7fae200754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 >> 172.21.15.38:0/3593373139 conn(0x7fae200fcd10 msgr2=0x7fae20105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:55.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 shutdown_connections 2024-04-22T01:12:55.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.259+0000 7fae24d25640 1 -- 172.21.15.38:0/3593373139 wait complete. 2024-04-22T01:12:55.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 -- 172.21.15.38:0/4249078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e0100ec0 msgr2=0x7f75e01032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 --2- 172.21.15.38:0/4249078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e0100ec0 0x7f75e01032b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009ca0 tx=0x7f75d002c180 comp rx=0 tx=0).stop 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 -- 172.21.15.38:0/4249078294 shutdown_connections 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 --2- 172.21.15.38:0/4249078294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e01062f0 0x7f75e01087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 --2- 172.21.15.38:0/4249078294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e01037f0 0x7f75e0105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 --2- 172.21.15.38:0/4249078294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e0100ec0 0x7f75e01032b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 -- 172.21.15.38:0/4249078294 >> 172.21.15.38:0/4249078294 conn(0x7f75e00faf40 msgr2=0x7f75e00fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 -- 172.21.15.38:0/4249078294 shutdown_connections 2024-04-22T01:12:55.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.374+0000 7f75e49d2640 1 -- 172.21.15.38:0/4249078294 wait complete. 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 Processor -- start 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 -- start start 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e0100ec0 0x7f75e0100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e01037f0 0x7f75e019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 0x7f75e019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010c320 con 0x7f75e01062f0 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010c1a0 con 0x7f75e01037f0 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75e49d2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010c4a0 con 0x7f75e0100ec0 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75ded9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 0x7f75e019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75ded9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 0x7f75e019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41172/0 (socket says 172.21.15.38:41172) 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.375+0000 7f75ded9d640 1 -- 172.21.15.38:0/2845307479 learned_addr learned my addr 172.21.15.38:0/2845307479 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ddd9b640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e01037f0 0x7f75e019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75de59c640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e0100ec0 0x7f75e0100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ded9d640 1 -- 172.21.15.38:0/2845307479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e0100ec0 msgr2=0x7f75e0100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ded9d640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e0100ec0 0x7f75e0100740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ded9d640 1 -- 172.21.15.38:0/2845307479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e01037f0 msgr2=0x7f75e019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ded9d640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e01037f0 0x7f75e019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ded9d640 1 -- 172.21.15.38:0/2845307479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e019f770 con 0x7f75e01062f0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75ded9d640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 0x7f75e019ee60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f75d800d750 tx=0x7f75d800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d8017070 con 0x7f75e01062f0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d8005500 con 0x7f75e01062f0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e019fa00 con 0x7f75e01062f0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d8004d90 con 0x7f75e01062f0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.376+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e019ff50 con 0x7f75e01062f0 2024-04-22T01:12:55.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.377+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e00fca30 con 0x7f75e01062f0 2024-04-22T01:12:55.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.378+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f75d8004f30 con 0x7f75e01062f0 2024-04-22T01:12:55.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.379+0000 7f75d77fe640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f75cc0a2b90 0x7f75cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.379+0000 7f75de59c640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f75cc0a2b90 0x7f75cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.379+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f75d80cee20 con 0x7f75e01062f0 2024-04-22T01:12:55.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.380+0000 7f75de59c640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f75cc0a2b90 0x7f75cc0a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006010 tx=0x7f75d0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:55.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.384+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f75d809b970 con 0x7f75e01062f0 2024-04-22T01:12:55.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.569+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f75e0062340 con 0x7f75e01062f0 2024-04-22T01:12:55.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.569+0000 7f75d77fe640 1 -- 172.21.15.38:0/2845307479 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f75d809b270 con 0x7f75e01062f0 2024-04-22T01:12:55.570 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:12:55.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f75cc0a2b90 msgr2=0x7f75cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f75cc0a2b90 0x7f75cc0a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006010 tx=0x7f75d0000f40 comp rx=0 tx=0).stop 2024-04-22T01:12:55.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 msgr2=0x7f75e019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 0x7f75e019ee60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f75d800d750 tx=0x7f75d800dc10 comp rx=0 tx=0).stop 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 shutdown_connections 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f75cc0a2b90 0x7f75cc0a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75e01062f0 0x7f75e019ee60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e01037f0 0x7f75e019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 --2- 172.21.15.38:0/2845307479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e0100ec0 0x7f75e0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 >> 172.21.15.38:0/2845307479 conn(0x7f75e00faf40 msgr2=0x7f75e0101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 shutdown_connections 2024-04-22T01:12:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.571+0000 7f75e49d2640 1 -- 172.21.15.38:0/2845307479 wait complete. 2024-04-22T01:12:55.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 -- 172.21.15.38:0/2181794562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c107460 msgr2=0x7f8d3c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 --2- 172.21.15.38:0/2181794562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c107460 0x7f8d3c1098b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009ca0 tx=0x7f8d3402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 -- 172.21.15.38:0/2181794562 shutdown_connections 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 --2- 172.21.15.38:0/2181794562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c107460 0x7f8d3c1098b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 --2- 172.21.15.38:0/2181794562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c104b30 0x7f8d3c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 --2- 172.21.15.38:0/2181794562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3c102200 0x7f8d3c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 -- 172.21.15.38:0/2181794562 >> 172.21.15.38:0/2181794562 conn(0x7f8d3c0fbbf0 msgr2=0x7f8d3c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 -- 172.21.15.38:0/2181794562 shutdown_connections 2024-04-22T01:12:55.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.692+0000 7f8d4108d640 1 -- 172.21.15.38:0/2181794562 wait complete. 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 Processor -- start 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 -- start start 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c102200 0x7f8d3c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 0x7f8d3c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3c107460 0x7f8d3c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c10d420 con 0x7f8d3c104b30 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c10d2a0 con 0x7f8d3c102200 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d4108d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c10d5a0 con 0x7f8d3c107460 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d33fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 0x7f8d3c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d33fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 0x7f8d3c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41186/0 (socket says 172.21.15.38:41186) 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d33fff640 1 -- 172.21.15.38:0/3589997500 learned_addr learned my addr 172.21.15.38:0/3589997500 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.693+0000 7f8d3b59e640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3c107460 0x7f8d3c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d3ad9d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c102200 0x7f8d3c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d33fff640 1 -- 172.21.15.38:0/3589997500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3c107460 msgr2=0x7f8d3c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d33fff640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3c107460 0x7f8d3c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d33fff640 1 -- 172.21.15.38:0/3589997500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c102200 msgr2=0x7f8d3c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d33fff640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c102200 0x7f8d3c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d33fff640 1 -- 172.21.15.38:0/3589997500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3c19f6b0 con 0x7f8d3c104b30 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d3ad9d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c102200 0x7f8d3c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d33fff640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 0x7f8d3c197fd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400e9c0 tx=0x7f8d2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400cd80 con 0x7f8d3c104b30 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d24005500 con 0x7f8d3c104b30 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3c19f9a0 con 0x7f8d3c104b30 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24010690 con 0x7f8d3c104b30 2024-04-22T01:12:55.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.694+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d3c19fef0 con 0x7f8d3c104b30 2024-04-22T01:12:55.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.695+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d3c0fd6e0 con 0x7f8d3c104b30 2024-04-22T01:12:55.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.696+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8d240046c0 con 0x7f8d3c104b30 2024-04-22T01:12:55.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.697+0000 7f8d38d99640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d280a2b90 0x7f8d280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:55.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.697+0000 7f8d3ad9d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d280a2b90 0x7f8d280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:55.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.697+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f8d240cf860 con 0x7f8d3c104b30 2024-04-22T01:12:55.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.697+0000 7f8d3ad9d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d280a2b90 0x7f8d280a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009c20 tx=0x7f8d2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:55.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.701+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8d2409c3f0 con 0x7f8d3c104b30 2024-04-22T01:12:55.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.887+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8d3c062340 con 0x7f8d3c104b30 2024-04-22T01:12:55.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.887+0000 7f8d38d99640 1 -- 172.21.15.38:0/3589997500 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f8d2409bcf0 con 0x7f8d3c104b30 2024-04-22T01:12:55.888 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:12:55.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d280a2b90 msgr2=0x7f8d280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d280a2b90 0x7f8d280a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009c20 tx=0x7f8d2c005f90 comp rx=0 tx=0).stop 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 msgr2=0x7f8d3c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 0x7f8d3c197fd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400e9c0 tx=0x7f8d2400ee80 comp rx=0 tx=0).stop 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 shutdown_connections 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8d280a2b90 0x7f8d280a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3c107460 0x7f8d3c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d3c104b30 0x7f8d3c197fd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 --2- 172.21.15.38:0/3589997500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c102200 0x7f8d3c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 >> 172.21.15.38:0/3589997500 conn(0x7f8d3c0fbbf0 msgr2=0x7f8d3c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 shutdown_connections 2024-04-22T01:12:55.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:55.890+0000 7f8d4108d640 1 -- 172.21.15.38:0/3589997500 wait complete. 2024-04-22T01:12:55.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:55 smithi038 ceph-mon[28043]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:55.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:55 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2079233188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:55.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:55 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3593373139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:55.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:55 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2845307479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:55.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 -- 172.21.15.38:0/1642664870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a81006a0 msgr2=0x7f73a8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 --2- 172.21.15.38:0/1642664870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a81006a0 0x7f73a8100ac0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f739c009900 tx=0x7f739c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 -- 172.21.15.38:0/1642664870 shutdown_connections 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 --2- 172.21.15.38:0/1642664870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8102250 0x7f73a8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 --2- 172.21.15.38:0/1642664870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a8101890 0x7f73a8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 --2- 172.21.15.38:0/1642664870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a81006a0 0x7f73a8100ac0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 -- 172.21.15.38:0/1642664870 >> 172.21.15.38:0/1642664870 conn(0x7f73a80fbc30 msgr2=0x7f73a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 -- 172.21.15.38:0/1642664870 shutdown_connections 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.003+0000 7f73af03b640 1 -- 172.21.15.38:0/1642664870 wait complete. 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 Processor -- start 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 -- start start 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a81006a0 0x7f73a8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8101890 0x7f73a8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 0x7f73a819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f73a810c370 con 0x7f73a8102250 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f73a810c1f0 con 0x7f73a81006a0 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73af03b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73a810c4f0 con 0x7f73a8101890 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73ad5d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 0x7f73a819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73ad5d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 0x7f73a819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41198/0 (socket says 172.21.15.38:41198) 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73ad5d8640 1 -- 172.21.15.38:0/698316082 learned_addr learned my addr 172.21.15.38:0/698316082 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.004+0000 7f73acdd7640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a81006a0 0x7f73a8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73ad5d8640 1 -- 172.21.15.38:0/698316082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8101890 msgr2=0x7f73a8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73ad5d8640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8101890 0x7f73a8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73ad5d8640 1 -- 172.21.15.38:0/698316082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a81006a0 msgr2=0x7f73a8197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73ad5d8640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a81006a0 0x7f73a8197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73ad5d8640 1 -- 172.21.15.38:0/698316082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a819f6b0 con 0x7f73a8102250 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73acdd7640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a81006a0 0x7f73a8197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73ad5d8640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 0x7f73a819f040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f73a000eb10 tx=0x7f73a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0004b20 con 0x7f73a8102250 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73a0004cc0 con 0x7f73a8102250 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a819f9a0 con 0x7f73a8102250 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0010760 con 0x7f73a8102250 2024-04-22T01:12:56.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.005+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a819ff40 con 0x7f73a8102250 2024-04-22T01:12:56.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.006+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a81a01d0 con 0x7f73a8102250 2024-04-22T01:12:56.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.007+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f73a0005020 con 0x7f73a8102250 2024-04-22T01:12:56.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.008+0000 7f73a5ffb640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73980a2c10 0x7f73980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.008+0000 7f73acdd7640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73980a2c10 0x7f73980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.008+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f73a00cfb40 con 0x7f73a8102250 2024-04-22T01:12:56.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.008+0000 7f73acdd7640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73980a2c10 0x7f73980a50d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f739c02c170 tx=0x7f739c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.012+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f73a009c690 con 0x7f73a8102250 2024-04-22T01:12:56.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:55 smithi203 ceph-mon[31825]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:55 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2079233188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:55 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3593373139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:55 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2845307479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:55 smithi111 ceph-mon[31057]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:12:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:55 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2079233188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:55 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3593373139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:55 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2845307479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.236+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73a81a03a0 con 0x7f73a8102250 2024-04-22T01:12:56.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.237+0000 7f73a5ffb640 1 -- 172.21.15.38:0/698316082 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f73a009bf90 con 0x7f73a8102250 2024-04-22T01:12:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:56.237 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:11:00.429607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24451,24472,24446,24527,24485]},"id":1}]} 2024-04-22T01:12:56.238 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T01:12:56.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73980a2c10 msgr2=0x7f73980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73980a2c10 0x7f73980a50d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f739c02c170 tx=0x7f739c02df90 comp rx=0 tx=0).stop 2024-04-22T01:12:56.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 msgr2=0x7f73a819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 0x7f73a819f040 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f73a000eb10 tx=0x7f73a000c2d0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 shutdown_connections 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f73980a2c10 0x7f73980a50d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73a8102250 0x7f73a819f040 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8101890 0x7f73a8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 --2- 172.21.15.38:0/698316082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73a81006a0 0x7f73a8197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 >> 172.21.15.38:0/698316082 conn(0x7f73a80fbc30 msgr2=0x7f73a80fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 shutdown_connections 2024-04-22T01:12:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.239+0000 7f73af03b640 1 -- 172.21.15.38:0/698316082 wait complete. 2024-04-22T01:12:56.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 -- 172.21.15.38:0/702774903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b4101890 msgr2=0x7fc0b4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 --2- 172.21.15.38:0/702774903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b4101890 0x7fc0b4101d10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009900 tx=0x7fc0a802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 -- 172.21.15.38:0/702774903 shutdown_connections 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 --2- 172.21.15.38:0/702774903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0b4102250 0x7fc0b4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 --2- 172.21.15.38:0/702774903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b4101890 0x7fc0b4101d10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 --2- 172.21.15.38:0/702774903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b41006a0 0x7fc0b4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 -- 172.21.15.38:0/702774903 >> 172.21.15.38:0/702774903 conn(0x7fc0b40fbc30 msgr2=0x7fc0b40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 -- 172.21.15.38:0/702774903 shutdown_connections 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.353+0000 7fc0bba40640 1 -- 172.21.15.38:0/702774903 wait complete. 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 Processor -- start 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 -- start start 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 0x7fc0b410d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101890 0x7fc0b410ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0b4102250 0x7fc0b410bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b406a7f0 con 0x7fc0b41006a0 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b406a670 con 0x7fc0b4101890 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0bba40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b406a970 con 0x7fc0b4102250 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0b97dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 0x7fc0b410d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0b97dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 0x7fc0b410d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41212/0 (socket says 172.21.15.38:41212) 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0b97dc640 1 -- 172.21.15.38:0/3174552698 learned_addr learned my addr 172.21.15.38:0/3174552698 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0b9fdd640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0b4102250 0x7fc0b410bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.354+0000 7fc0b8fdb640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101890 0x7fc0b410ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b97dc640 1 -- 172.21.15.38:0/3174552698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0b4102250 msgr2=0x7fc0b410bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b97dc640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0b4102250 0x7fc0b410bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b97dc640 1 -- 172.21.15.38:0/3174552698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101890 msgr2=0x7fc0b410ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b97dc640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101890 0x7fc0b410ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b97dc640 1 -- 172.21.15.38:0/3174552698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b410c860 con 0x7fc0b41006a0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b97dc640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 0x7fc0b410d450 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400ea80 tx=0x7fc0a400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a400ce40 con 0x7fc0b41006a0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b41a1bb0 con 0x7fc0b41006a0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0a4011950 con 0x7fc0b41006a0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a401ab80 con 0x7fc0b41006a0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.355+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0b41a2120 con 0x7fc0b41006a0 2024-04-22T01:12:56.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.356+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0b4107a40 con 0x7fc0b41006a0 2024-04-22T01:12:56.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.357+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc0a4011b00 con 0x7fc0b41006a0 2024-04-22T01:12:56.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.358+0000 7fc0b27fc640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc09c0a2b90 0x7fc09c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.358+0000 7fc0b8fdb640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc09c0a2b90 0x7fc09c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.358+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fc0a40cff20 con 0x7fc0b41006a0 2024-04-22T01:12:56.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.358+0000 7fc0b8fdb640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc09c0a2b90 0x7fc09c0a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8006010 tx=0x7fc0a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.361+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc0a4016030 con 0x7fc0b41006a0 2024-04-22T01:12:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.550+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0b402d340 con 0x7fc0b41006a0 2024-04-22T01:12:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.551+0000 7fc0b27fc640 1 -- 172.21.15.38:0/3174552698 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7fc0a409cab0 con 0x7fc0b41006a0 2024-04-22T01:12:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:12:56.552 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.553+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc09c0a2b90 msgr2=0x7fc09c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc09c0a2b90 0x7fc09c0a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8006010 tx=0x7fc0a8000bc0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 msgr2=0x7fc0b410d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 0x7fc0b410d450 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400ea80 tx=0x7fc0a400ef40 comp rx=0 tx=0).stop 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 shutdown_connections 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc09c0a2b90 0x7fc09c0a5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0b4102250 0x7fc0b410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101890 0x7fc0b410ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 --2- 172.21.15.38:0/3174552698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0b41006a0 0x7fc0b410d450 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 >> 172.21.15.38:0/3174552698 conn(0x7fc0b40fbc30 msgr2=0x7fc0b4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 shutdown_connections 2024-04-22T01:12:56.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.554+0000 7fc0bba40640 1 -- 172.21.15.38:0/3174552698 wait complete. 2024-04-22T01:12:56.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T01:10:29.943110+0000', 'modified': '2024-04-22T01:11:00.429607+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24527, 'mds_1': 24485, 'mds_2': 24472, 'mds_3': 24451, 'mds_4': 24446}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14595': {'gid': 14595, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6839/1149927748', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 1149927748}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 1149927748}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14607': {'gid': 14607, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6841/3631879100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6840', 'nonce': 3631879100}, {'type': 'v1', 'addr': '172.21.15.38:6841', 'nonce': 3631879100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24400': {'gid': 24400, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/3645664141', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 3645664141}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 3645664141}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6833/1598672021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6832', 'nonce': 1598672021}, {'type': 'v1', 'addr': '172.21.15.111:6833', 'nonce': 1598672021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24439': {'gid': 24439, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/4027190739', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 4027190739}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 4027190739}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.111:6835/2761116669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 2761116669}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 2761116669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6837/2993816469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 2993816469}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 2993816469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.203:6839/765839756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 765839756}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 765839756}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/1636882248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 1636882248}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 1636882248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3001874105', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3001874105}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3001874105}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24451, 24472, 24446, 24527, 24485]} 2024-04-22T01:12:56.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T01:12:56.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 -- 172.21.15.38:0/181176840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7801006a0 msgr2=0x7ff780100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 --2- 172.21.15.38:0/181176840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7801006a0 0x7ff780100ac0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff770009900 tx=0x7ff77002bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 -- 172.21.15.38:0/181176840 shutdown_connections 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 --2- 172.21.15.38:0/181176840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff780102250 0x7ff780108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 --2- 172.21.15.38:0/181176840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff780101890 0x7ff780101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 --2- 172.21.15.38:0/181176840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7801006a0 0x7ff780100ac0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 -- 172.21.15.38:0/181176840 >> 172.21.15.38:0/181176840 conn(0x7ff7800fbc30 msgr2=0x7ff7800fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 -- 172.21.15.38:0/181176840 shutdown_connections 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.669+0000 7ff786597640 1 -- 172.21.15.38:0/181176840 wait complete. 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 Processor -- start 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 -- start start 2024-04-22T01:12:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7801006a0 0x7ff78006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff780101890 0x7ff78019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 0x7ff78019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff78010c320 con 0x7ff780102250 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff78010c1a0 con 0x7ff7801006a0 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff786597640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff78010c4a0 con 0x7ff780101890 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff784b34640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 0x7ff78019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff784b34640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 0x7ff78019eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41226/0 (socket says 172.21.15.38:41226) 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.670+0000 7ff784b34640 1 -- 172.21.15.38:0/3735781451 learned_addr learned my addr 172.21.15.38:0/3735781451 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff780101890 0x7ff78019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7801006a0 0x7ff78006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff784b34640 1 -- 172.21.15.38:0/3735781451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff780101890 msgr2=0x7ff78019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff784b34640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff780101890 0x7ff78019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff784b34640 1 -- 172.21.15.38:0/3735781451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7801006a0 msgr2=0x7ff78006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff784b34640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7801006a0 0x7ff78006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff784b34640 1 -- 172.21.15.38:0/3735781451 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78019f780 con 0x7ff780102250 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff784b34640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 0x7ff78019eec0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff77800cad0 tx=0x7ff77800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff77d7fa640 1 -- 172.21.15.38:0/3735781451 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77800ae20 con 0x7ff780102250 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff77d7fa640 1 -- 172.21.15.38:0/3735781451 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff778005020 con 0x7ff780102250 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78019fa70 con 0x7ff780102250 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff77d7fa640 1 -- 172.21.15.38:0/3735781451 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77800f6f0 con 0x7ff780102250 2024-04-22T01:12:56.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.671+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78019ff70 con 0x7ff780102250 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff77d7fa640 1 -- 172.21.15.38:0/3735781451 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff7780051d0 con 0x7ff780102250 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 Processor -- start 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 -- start start 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1070 0x55ab4e5c1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5c9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e525df0 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e4ce640 con 0x55ab4e5c1a60 2024-04-22T01:12:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff78917e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e521db0 con 0x55ab4e5c1070 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.674+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41238/0 (socket says 172.21.15.38:41238) 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3095118734 learned_addr learned my addr 172.21.15.38:0/3095118734 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1070 0x55ab4e5c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77d7fa640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7680a2b90 0x7ff7680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3095118734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1070 msgr2=0x55ab4e5c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1070 0x55ab4e5c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3095118734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 msgr2=0x55ab4e5c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff784b34640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7680a2b90 0x7ff7680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3095118734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab4e23a2d0 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77d7fa640 1 -- 172.21.15.38:0/3735781451 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff7780cf400 con 0x7ff780102250 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5c9060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff774011700 tx=0x7ff77400a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55e359502a972e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff7667fc640 1 -- 172.21.15.38:0/3095118734 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774004020 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff784b34640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7680a2b90 0x7ff7680a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff778098310 tx=0x7ff778066a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff7667fc640 1 -- 172.21.15.38:0/3095118734 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff774005500 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.675+0000 7ff7667fc640 1 -- 172.21.15.38:0/3095118734 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774004b40 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 -- 172.21.15.38:0/3095118734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 msgr2=0x55ab4e5c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5c9060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff774011700 tx=0x7ff77400a6a0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 -- 172.21.15.38:0/3095118734 shutdown_connections 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5c9060 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 --2- 172.21.15.38:0/3095118734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1070 0x55ab4e5c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 -- 172.21.15.38:0/3095118734 >> 172.21.15.38:0/3095118734 conn(0x55ab4e5bc5e0 msgr2=0x55ab4e5bc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 -- 172.21.15.38:0/3095118734 shutdown_connections 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 -- 172.21.15.38:0/3095118734 wait complete. 2024-04-22T01:12:56.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.676+0000 7ff78917e740 1 -- start start 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 Processor -- start 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5c9b80 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e4e1be0 con 0x55ab4e5c1070 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5c9d00 con 0x55ab4e5c1a60 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff78917e740 10 client.0 _ll_register_callbacks cb 0x55ab4e4e15d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41240/0 (socket says 172.21.15.38:41240) 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3607558046 learned_addr learned my addr 172.21.15.38:0/3607558046 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3607558046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 msgr2=0x55ab4e5cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 msgr2=0x55ab4e5cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3607558046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4e3046b0 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d2a50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff77400a640 tx=0x7ff77401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353ecb26861c684 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.677+0000 7ff75f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3607558046 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774004d10 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3607558046 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff77400f790 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff774005500 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3607558046 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff774032050 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 10 client.15411 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:56.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab4e5d59a0 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3607558046 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff774005500 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3607558046 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff77401e930 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 10 client.15411 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 10 client.15411 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 20 client.15411 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e5d7d20 0x55ab4e5d9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 4 client.15411 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab4e533330 con 0x55ab4e5d7d20 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.678+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e5d7d20 0x55ab4e5d9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.679+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e5d7d20 0x55ab4e5d9190 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d4e455166190d5c server_cookie=746cd337d78c75db in_seq=0 out_seq=0 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.679+0000 7ff75f7fe640 10 client.15411 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3607558046 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab4e533330 con 0x55ab4e5d7d20 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff75f7fe640 10 client.15411 handle_command_reply: tid=0 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 client.15411 shutdown 2024-04-22T01:12:56.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 msgr2=0x55ab4e5d2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d2a50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff77400a640 tx=0x7ff77401a040 comp rx=0 tx=0).stop 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 shutdown_connections 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e5d7d20 0x55ab4e5d9190 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d2a50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff75f7fe640 0 client.15411 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 --2- 172.21.15.38:0/3607558046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 >> 172.21.15.38:0/3607558046 conn(0x55ab4e5bc5e0 msgr2=0x55ab4e5bb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 shutdown_connections 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 1 -- 172.21.15.38:0/3607558046 wait complete. 2024-04-22T01:12:56.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.680+0000 7ff78917e740 20 client.15411 trim_cache size 0 max 16384 2024-04-22T01:12:56.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 Processor -- start 2024-04-22T01:12:56.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 -- start start 2024-04-22T01:12:56.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5d74d0 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5d7ba0 con 0x55ab4e5c1070 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff78917e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5d8270 con 0x55ab4e5c1a60 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50820/0 (socket says 172.21.15.38:50820) 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff77f7fe640 1 -- 172.21.15.38:0/2997623568 learned_addr learned my addr 172.21.15.38:0/2997623568 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff77ffff640 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.699+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77ffff640 1 -- 172.21.15.38:0/2997623568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 msgr2=0x55ab4e5cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77ffff640 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77ffff640 1 -- 172.21.15.38:0/2997623568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 msgr2=0x55ab4e5d6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77ffff640 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77ffff640 1 -- 172.21.15.38:0/2997623568 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab4e5c1e40 con 0x55ab4e5c1a60 2024-04-22T01:12:56.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:12:56.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff77ffff640 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5d2970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff770036040 tx=0x7ff770009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2703d1a23b612a25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff765ffb640 1 -- 172.21.15.38:0/2997623568 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77004b070 con 0x55ab4e5c1a60 2024-04-22T01:12:56.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff765ffb640 1 -- 172.21.15.38:0/2997623568 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff770046bc0 con 0x55ab4e5c1a60 2024-04-22T01:12:56.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.700+0000 7ff765ffb640 1 -- 172.21.15.38:0/2997623568 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff770046660 con 0x55ab4e5c1a60 2024-04-22T01:12:56.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 -- 172.21.15.38:0/2997623568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 msgr2=0x55ab4e5d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5d2970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff770036040 tx=0x7ff770009260 comp rx=0 tx=0).stop 2024-04-22T01:12:56.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 -- 172.21.15.38:0/2997623568 shutdown_connections 2024-04-22T01:12:56.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5d2970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 --2- 172.21.15.38:0/2997623568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1070 0x55ab4e5cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 -- 172.21.15.38:0/2997623568 >> 172.21.15.38:0/2997623568 conn(0x55ab4e5bc5e0 msgr2=0x55ab4e5b7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 -- 172.21.15.38:0/2997623568 shutdown_connections 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 -- 172.21.15.38:0/2997623568 wait complete. 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 -- start start 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.701+0000 7ff78917e740 1 Processor -- start 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 0x55ab4e5ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5d9010 con 0x55ab4e5c1070 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5d8940 con 0x55ab4e5c1a60 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab4e5d96e0 con 0x55ab4e5c6bd0 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff78917e740 10 client.0 _ll_register_callbacks cb 0x7ff774004920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 0x55ab4e5ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 0x55ab4e5ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50826/0 (socket says 172.21.15.38:50826) 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3752777415 learned_addr learned my addr 172.21.15.38:0/3752777415 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3752777415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c6bd0 msgr2=0x55ab4e5d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 msgr2=0x55ab4e5cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 -- 172.21.15.38:0/3752777415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4e5d21f0 con 0x55ab4e5c1070 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff77f7fe640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 0x55ab4e5ce6a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff774011fd0 tx=0x7ff7740056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ff1cfc959648d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff75f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.702+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3752777415 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77403d020 con 0x55ab4e5c1070 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3752777415 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff77400a640 con 0x55ab4e5c1070 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff770046bc0 con 0x55ab4e5c1070 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3752777415 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff774045050 con 0x55ab4e5c1070 2024-04-22T01:12:56.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 10 client.15417 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab4e63e2b0 con 0x55ab4e5c1070 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3752777415 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff770046bc0 con 0x55ab4e5c1070 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3752777415 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff77400fad0 con 0x55ab4e5c1070 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 10 client.15417 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 10 client.15417 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 20 client.15417 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e642ec0 0x55ab4e641bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 4 client.15417 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.703+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "status"}) -- 0x55ab4e52d680 con 0x55ab4e642ec0 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e642ec0 0x55ab4e641bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff77ffff640 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e642ec0 0x55ab4e641bd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1dd8210e3964dd5 server_cookie=14f489cfb1359d46 in_seq=0 out_seq=0 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff75f7fe640 10 client.15417 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff75f7fe640 1 -- 172.21.15.38:0/3752777415 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ab4e52d680 con 0x55ab4e642ec0 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff75f7fe640 10 client.15417 handle_command_reply: tid=0 2024-04-22T01:12:56.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 client.15417 shutdown 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 msgr2=0x55ab4e5ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 0x55ab4e5ce6a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff774011fd0 tx=0x7ff7740056b0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 shutdown_connections 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55ab4e642ec0 0x55ab4e641bd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab4e5c6bd0 0x55ab4e5d1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab4e5c1a60 0x55ab4e5cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 --2- 172.21.15.38:0/3752777415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab4e5c1070 0x55ab4e5ce6a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff75f7fe640 0 client.15417 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 >> 172.21.15.38:0/3752777415 conn(0x55ab4e5bc5e0 msgr2=0x55ab4e5d6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.704+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 shutdown_connections 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff78917e740 1 -- 172.21.15.38:0/3752777415 wait complete. 2024-04-22T01:12:56.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff78917e740 20 client.15417 trim_cache size 0 max 16384 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 4, 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24446, 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 116.787600367, 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 170.55784827700001 2024-04-22T01:12:56.711 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7680a2b90 msgr2=0x7ff7680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7680a2b90 0x7ff7680a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff778098310 tx=0x7ff778066a60 comp rx=0 tx=0).stop 2024-04-22T01:12:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 msgr2=0x7ff78019eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 0x7ff78019eec0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff77800cad0 tx=0x7ff77800cf90 comp rx=0 tx=0).stop 2024-04-22T01:12:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 shutdown_connections 2024-04-22T01:12:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7680a2b90 0x7ff7680a5050 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff780102250 0x7ff78019eec0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff780101890 0x7ff78019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 --2- 172.21.15.38:0/3735781451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7801006a0 0x7ff78006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 >> 172.21.15.38:0/3735781451 conn(0x7ff7800fbc30 msgr2=0x7ff780104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 shutdown_connections 2024-04-22T01:12:56.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.705+0000 7ff786597640 1 -- 172.21.15.38:0/3735781451 wait complete. 2024-04-22T01:12:56.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T01:12:56.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 -- 172.21.15.38:0/889630238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90104b30 msgr2=0x7f6b90106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/889630238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90104b30 0x7f6b90106f20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009900 tx=0x7f6b7802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 -- 172.21.15.38:0/889630238 shutdown_connections 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/889630238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 0x7f6b901098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/889630238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90104b30 0x7f6b90106f20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/889630238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90102200 0x7f6b901045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 -- 172.21.15.38:0/889630238 >> 172.21.15.38:0/889630238 conn(0x7f6b900fbbf0 msgr2=0x7f6b900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 -- 172.21.15.38:0/889630238 shutdown_connections 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 -- 172.21.15.38:0/889630238 wait complete. 2024-04-22T01:12:56.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.819+0000 7f6b94b4f640 1 Processor -- start 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 -- start start 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 0x7f6b90197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90104b30 0x7f6b90198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 0x7f6b9019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010d410 con 0x7f6b90102200 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010d290 con 0x7f6b90107460 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b94b4f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010d590 con 0x7f6b90104b30 2024-04-22T01:12:56.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 0x7f6b90197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 0x7f6b90197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50840/0 (socket says 172.21.15.38:50840) 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 -- 172.21.15.38:0/2854428486 learned_addr learned my addr 172.21.15.38:0/2854428486 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90104b30 0x7f6b90198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 0x7f6b9019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 -- 172.21.15.38:0/2854428486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90104b30 msgr2=0x7f6b90198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90104b30 0x7f6b90198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 -- 172.21.15.38:0/2854428486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 msgr2=0x7f6b9019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 0x7f6b9019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 -- 172.21.15.38:0/2854428486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9019f710 con 0x7f6b90102200 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90104b30 0x7f6b90198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 0x7f6b9019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.820+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 0x7f6b90197af0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000d770 tx=0x7f6b8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.821+0000 7f6b87fff640 1 -- 172.21.15.38:0/2854428486 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80017070 con 0x7f6b90102200 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.821+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9019fa00 con 0x7f6b90102200 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.821+0000 7f6b87fff640 1 -- 172.21.15.38:0/2854428486 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b80005500 con 0x7f6b90102200 2024-04-22T01:12:56.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.821+0000 7f6b87fff640 1 -- 172.21.15.38:0/2854428486 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80004e00 con 0x7f6b90102200 2024-04-22T01:12:56.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.821+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9019fed0 con 0x7f6b90102200 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.823+0000 7f6b87fff640 1 -- 172.21.15.38:0/2854428486 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6b80004fa0 con 0x7f6b90102200 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.823+0000 7f6b97813740 1 Processor -- start 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.823+0000 7f6b97813740 1 -- start start 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.823+0000 7f6b97813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 0x55ca0ef28490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b97813740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef2d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b97813740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef30060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b97813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ee8cdf0 con 0x55ca0ef28070 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b97813740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ee35640 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b97813740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ee88db0 con 0x55ca0ef28a60 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 0x55ca0ef28490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 0x55ca0ef28490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50852/0 (socket says 172.21.15.38:50852) 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2584598315 learned_addr learned my addr 172.21.15.38:0/2584598315 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b87fff640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6b7c0a2b90 0x7f6b7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef30060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef2d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2584598315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 msgr2=0x55ca0ef2d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef2d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2584598315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef2dbd0 msgr2=0x55ca0ef30060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef30060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2584598315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca0eba12d0 con 0x55ca0ef28070 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef30060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b87fff640 1 -- 172.21.15.38:0/2854428486 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6b800cef50 con 0x7f6b90102200 2024-04-22T01:12:56.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6b7c0a2b90 0x7f6b7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 0x55ca0ef28490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800f390 tx=0x7f6b8800f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75867599c70d8d50 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.824+0000 7f6b84d99640 1 -- 172.21.15.38:0/2584598315 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8801a610 con 0x55ca0ef28070 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b84d99640 1 -- 172.21.15.38:0/2584598315 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b8801ac90 con 0x55ca0ef28070 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b84d99640 1 -- 172.21.15.38:0/2584598315 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88010a20 con 0x55ca0ef28070 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 -- 172.21.15.38:0/2584598315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 msgr2=0x55ca0ef28490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 0x55ca0ef28490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800f390 tx=0x7f6b8800f850 comp rx=0 tx=0).stop 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6b7c0a2b90 0x7f6b7c0a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6b88020000 tx=0x7f6b8801d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 -- 172.21.15.38:0/2584598315 shutdown_connections 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef30060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef2d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 --2- 172.21.15.38:0/2584598315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28070 0x55ca0ef28490 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 -- 172.21.15.38:0/2584598315 >> 172.21.15.38:0/2584598315 conn(0x55ca0ef235e0 msgr2=0x55ca0ef239e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 -- 172.21.15.38:0/2584598315 shutdown_connections 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.825+0000 7f6b97813740 1 -- 172.21.15.38:0/2584598315 wait complete. 2024-04-22T01:12:56.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 -- start start 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 Processor -- start 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef32ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef39ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef30b80 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ee48be0 con 0x55ca0ef28070 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef30d00 con 0x55ca0ef28a60 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b97813740 10 client.0 _ll_register_callbacks cb 0x55ca0ee485d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef39ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef39ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50862/0 (socket says 172.21.15.38:50862) 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2087011527 learned_addr learned my addr 172.21.15.38:0/2087011527 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 msgr2=0x55ca0ef32ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef32ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2087011527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 msgr2=0x55ca0ef325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef325a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.826+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/2087011527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca0ec6b6b0 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef39ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6b880124f0 tx=0x7f6b88010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a99dc284c247546 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b75ffb640 1 -- 172.21.15.38:0/2087011527 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88033060 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b75ffb640 1 -- 172.21.15.38:0/2087011527 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6b8801a410 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b8801ac90 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b75ffb640 1 -- 172.21.15.38:0/2087011527 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6b8803a050 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 10 client.15429 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ca0ef3cb00 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b75ffb640 1 -- 172.21.15.38:0/2087011527 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b8801ac90 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b75ffb640 1 -- 172.21.15.38:0/2087011527 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6b8802ea60 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 10 client.15429 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 10 client.15429 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:12:56.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 20 client.15429 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0ef3ee30 0x55ca0ef402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 4 client.15429 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.827+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca0ee9a330 con 0x55ca0ef3ee30 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.828+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0ef3ee30 0x55ca0ef402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.828+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0ef3ee30 0x55ca0ef402a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8b3fc0b3600be8c4 server_cookie=90200d1e0ffc840d in_seq=0 out_seq=0 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.828+0000 7f6b75ffb640 10 client.15429 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b75ffb640 1 -- 172.21.15.38:0/2087011527 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca0ee9a330 con 0x55ca0ef3ee30 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b75ffb640 10 client.15429 handle_command_reply: tid=0 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 client.15429 shutdown 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 msgr2=0x55ca0ef39ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef39ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6b880124f0 tx=0x7f6b88010f70 comp rx=0 tx=0).stop 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 shutdown_connections 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0ef3ee30 0x55ca0ef402a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef39ab0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef32ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b75ffb640 0 client.15429 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 --2- 172.21.15.38:0/2087011527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 >> 172.21.15.38:0/2087011527 conn(0x55ca0ef235e0 msgr2=0x55ca0ef22720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 shutdown_connections 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.829+0000 7f6b97813740 1 -- 172.21.15.38:0/2087011527 wait complete. 2024-04-22T01:12:56.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.830+0000 7f6b97813740 20 client.15429 trim_cache size 0 max 16384 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 Processor -- start 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 -- start start 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef32a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 0x55ca0ef399d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef239c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef3e5e0 con 0x55ca0ef28a60 2024-04-22T01:12:56.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef3ecb0 con 0x55ca0ef28070 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b97813740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef3f380 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 0x55ca0ef399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 0x55ca0ef399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50864/0 (socket says 172.21.15.38:50864) 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 -- 172.21.15.38:0/3922815257 learned_addr learned my addr 172.21.15.38:0/3922815257 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef32a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 -- 172.21.15.38:0/3922815257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef2dbd0 msgr2=0x55ca0ef239c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef239c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 -- 172.21.15.38:0/3922815257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 msgr2=0x55ca0ef32a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef32a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 -- 172.21.15.38:0/3922815257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca0ef21c20 con 0x55ca0ef28a60 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef32a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.848+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 0x55ca0ef399d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6b78005ab0 tx=0x7f6b78005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6cca6f770c00c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b77fff640 1 -- 172.21.15.38:0/3922815257 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7802ccc0 con 0x55ca0ef28a60 2024-04-22T01:12:56.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b77fff640 1 -- 172.21.15.38:0/3922815257 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b78005040 con 0x55ca0ef28a60 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b77fff640 1 -- 172.21.15.38:0/3922815257 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78004a40 con 0x55ca0ef28a60 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 -- 172.21.15.38:0/3922815257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 msgr2=0x55ca0ef399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 0x55ca0ef399d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6b78005ab0 tx=0x7f6b78005d50 comp rx=0 tx=0).stop 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 -- 172.21.15.38:0/3922815257 shutdown_connections 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef239c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef28a60 0x55ca0ef399d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 --2- 172.21.15.38:0/3922815257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef32a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 -- 172.21.15.38:0/3922815257 >> 172.21.15.38:0/3922815257 conn(0x55ca0ef235e0 msgr2=0x55ca0ef310d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 -- 172.21.15.38:0/3922815257 shutdown_connections 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 -- 172.21.15.38:0/3922815257 wait complete. 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 -- start start 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.849+0000 7f6b97813740 1 Processor -- start 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef37c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef38150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef38730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef40120 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef3fa50 con 0x55ca0ef28070 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ca0ef407f0 con 0x55ca0ef28a60 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 10 client.0 _ll_register_callbacks cb 0x7f6b880126b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef38730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef38730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50868/0 (socket says 172.21.15.38:50868) 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/1614686311 learned_addr learned my addr 172.21.15.38:0/1614686311 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef37c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b85d9b640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef38150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 msgr2=0x55ca0ef38150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef38150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/1614686311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 msgr2=0x55ca0ef37c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef37c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 -- 172.21.15.38:0/1614686311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca0ef38fb0 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef37c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b8ed9d640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef38730 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6b880093c0 tx=0x7f6b88012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970f213de864bc6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b75ffb640 1 -- 172.21.15.38:0/1614686311 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88043060 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b75ffb640 1 -- 172.21.15.38:0/1614686311 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6b88025860 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.850+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b78005040 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b75ffb640 1 -- 172.21.15.38:0/1614686311 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6b8804a050 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 10 client.15435 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ca0efa5120 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b75ffb640 1 -- 172.21.15.38:0/1614686311 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b78005040 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b75ffb640 1 -- 172.21.15.38:0/1614686311 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6b8802e6b0 con 0x55ca0ef2dbd0 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 10 client.15435 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:56.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 10 client.15435 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 20 client.15435 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0efa9c90 0x55ca0efa89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 4 client.15435 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "status"}) -- 0x55ca0ee94680 con 0x55ca0efa9c90 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.851+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0efa9c90 0x55ca0efa89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b75ffb640 10 client.15435 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b8e59c640 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0efa9c90 0x55ca0efa89a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa8df0d2ef465221 server_cookie=53d6fdaebff73e2f in_seq=0 out_seq=0 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b75ffb640 1 -- 172.21.15.38:0/1614686311 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ca0ee94680 con 0x55ca0efa9c90 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b75ffb640 10 client.15435 handle_command_reply: tid=0 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 client.15435 shutdown 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 msgr2=0x55ca0ef38730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef38730 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6b880093c0 tx=0x7f6b88012760 comp rx=0 tx=0).stop 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 shutdown_connections 2024-04-22T01:12:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55ca0efa9c90 0x55ca0efa89a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca0ef2dbd0 0x55ca0ef38730 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca0ef28a60 0x55ca0ef38150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 --2- 172.21.15.38:0/1614686311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ca0ef28070 0x55ca0ef37c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b75ffb640 0 client.15435 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 >> 172.21.15.38:0/1614686311 conn(0x55ca0ef235e0 msgr2=0x55ca0ef239c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 shutdown_connections 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 1 -- 172.21.15.38:0/1614686311 wait complete. 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.852+0000 7f6b97813740 20 client.15435 trim_cache size 0 max 16384 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 3, 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24451, 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 117.94909048300001, 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 165.081609762 2024-04-22T01:12:56.860 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6b7c0a2b90 msgr2=0x7f6b7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6b7c0a2b90 0x7f6b7c0a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6b88020000 tx=0x7f6b8801d000 comp rx=0 tx=0).stop 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 msgr2=0x7f6b90197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 0x7f6b90197af0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000d770 tx=0x7f6b8000dc30 comp rx=0 tx=0).stop 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 shutdown_connections 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6b7c0a2b90 0x7f6b7c0a5050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b90107460 0x7f6b9019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b90104b30 0x7f6b90198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 --2- 172.21.15.38:0/2854428486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b90102200 0x7f6b90197af0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 >> 172.21.15.38:0/2854428486 conn(0x7f6b900fbbf0 msgr2=0x7f6b90102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 shutdown_connections 2024-04-22T01:12:56.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.853+0000 7f6b94b4f640 1 -- 172.21.15.38:0/2854428486 wait complete. 2024-04-22T01:12:56.863 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3589997500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:56.863 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/698316082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:56.863 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3174552698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:56.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T01:12:56.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.967+0000 7f9ac82fb640 1 -- 172.21.15.38:0/705530321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0107460 msgr2=0x7f9ac01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.967+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/705530321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0107460 0x7f9ac01098b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009900 tx=0x7f9abc02bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- 172.21.15.38:0/705530321 shutdown_connections 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/705530321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0107460 0x7f9ac01098b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/705530321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ac0104b30 0x7f9ac0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/705530321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0102200 0x7f9ac01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- 172.21.15.38:0/705530321 >> 172.21.15.38:0/705530321 conn(0x7f9ac00fbbf0 msgr2=0x7f9ac00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- 172.21.15.38:0/705530321 shutdown_connections 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- 172.21.15.38:0/705530321 wait complete. 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 Processor -- start 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- start start 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 0x7f9ac0101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0104b30 0x7f9ac01001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ac0107460 0x7f9ac01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac010d420 con 0x7f9ac0102200 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac010d2a0 con 0x7f9ac0104b30 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac82fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac010d5a0 con 0x7f9ac0107460 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac6097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 0x7f9ac0101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac6097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 0x7f9ac0101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50884/0 (socket says 172.21.15.38:50884) 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.968+0000 7f9ac6097640 1 -- 172.21.15.38:0/504198720 learned_addr learned my addr 172.21.15.38:0/504198720 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac5896640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0104b30 0x7f9ac01001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac6097640 1 -- 172.21.15.38:0/504198720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ac0107460 msgr2=0x7f9ac01006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac6097640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ac0107460 0x7f9ac01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac6097640 1 -- 172.21.15.38:0/504198720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0104b30 msgr2=0x7f9ac01001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac6097640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0104b30 0x7f9ac01001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac6097640 1 -- 172.21.15.38:0/504198720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac0100f80 con 0x7f9ac0102200 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac5896640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0104b30 0x7f9ac01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac6097640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 0x7f9ac0101b70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800eab0 tx=0x7f9ab800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ab77fe640 1 -- 172.21.15.38:0/504198720 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800ce40 con 0x7f9ac0102200 2024-04-22T01:12:56.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ab77fe640 1 -- 172.21.15.38:0/504198720 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab8011950 con 0x7f9ac0102200 2024-04-22T01:12:56.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac01a1be0 con 0x7f9ac0102200 2024-04-22T01:12:56.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ab77fe640 1 -- 172.21.15.38:0/504198720 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8010ae0 con 0x7f9ac0102200 2024-04-22T01:12:56.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.969+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac01a2180 con 0x7f9ac0102200 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.971+0000 7f9ab77fe640 1 -- 172.21.15.38:0/504198720 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9ab8011b00 con 0x7f9ac0102200 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 Processor -- start 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 -- start start 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6dec560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6def060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ea6d4bdf0 con 0x560ea6de7070 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560ea6cf4640 con 0x560ea6de7a60 2024-04-22T01:12:56.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9acafc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560ea6d47db0 con 0x560ea6decbd0 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9ab77fe640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9aac0a2c10 0x7f9aac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9ac6898640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9ac5896640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6def060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9ac6898640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50886/0 (socket says 172.21.15.38:50886) 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9ac6898640 1 -- 172.21.15.38:0/2449530962 learned_addr learned my addr 172.21.15.38:0/2449530962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.972+0000 7f9ac6097640 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6dec560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac6898640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9aac0a2c10 0x7f9aac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ab77fe640 1 -- 172.21.15.38:0/504198720 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9ab80cfbf0 con 0x7f9ac0102200 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac5896640 1 -- 172.21.15.38:0/2449530962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 msgr2=0x560ea6dec560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac5896640 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6dec560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac5896640 1 -- 172.21.15.38:0/2449530962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 msgr2=0x560ea6de7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac5896640 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac5896640 1 -- 172.21.15.38:0/2449530962 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea6a602d0 con 0x560ea6decbd0 2024-04-22T01:12:56.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac6898640 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac5896640 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6def060 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00058c0 tx=0x7f9ab000bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8b69566333d1bbb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ac6898640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9aac0a2c10 0x7f9aac0a50d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9abc036040 tx=0x7f9abc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ab4bfe640 1 -- 172.21.15.38:0/2449530962 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0017420 con 0x560ea6decbd0 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ab4bfe640 1 -- 172.21.15.38:0/2449530962 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab0017aa0 con 0x560ea6decbd0 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.973+0000 7f9ab4bfe640 1 -- 172.21.15.38:0/2449530962 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000fae0 con 0x560ea6decbd0 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 -- 172.21.15.38:0/2449530962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 msgr2=0x560ea6def060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6def060 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00058c0 tx=0x7f9ab000bf40 comp rx=0 tx=0).stop 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 -- 172.21.15.38:0/2449530962 shutdown_connections 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6def060 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6dec560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 --2- 172.21.15.38:0/2449530962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 -- 172.21.15.38:0/2449530962 >> 172.21.15.38:0/2449530962 conn(0x560ea6de25e0 msgr2=0x560ea6de29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 -- 172.21.15.38:0/2449530962 shutdown_connections 2024-04-22T01:12:56.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 -- 172.21.15.38:0/2449530962 wait complete. 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.974+0000 7f9acafc1740 1 -- start start 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 Processor -- start 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7070 0x560ea6df1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6df1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 0x560ea6df8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ea6defb80 con 0x560ea6decbd0 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560ea6d07be0 con 0x560ea6de7070 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560ea6defd00 con 0x560ea6de7a60 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9acafc1740 10 client.0 _ll_register_callbacks cb 0x560ea6d075d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 0x560ea6df8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 0x560ea6df8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50896/0 (socket says 172.21.15.38:50896) 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 -- 172.21.15.38:0/504847125 learned_addr learned my addr 172.21.15.38:0/504847125 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac5896640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6df1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6097640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7070 0x560ea6df1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 -- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 msgr2=0x560ea6df1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6df1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 -- 172.21.15.38:0/504847125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7070 msgr2=0x560ea6df1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7070 0x560ea6df1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 -- 172.21.15.38:0/504847125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea6b2a6b0 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9ac6898640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 0x560ea6df8a80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9abc02c7b0 tx=0x7f9abc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb537205d9b54cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.975+0000 7f9aa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/504847125 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc004d10 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/504847125 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9abc0035f0 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ab0017aa0 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/504847125 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9abc059050 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 10 client.15447 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560ea6dfbad0 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/504847125 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab0017aa0 con 0x560ea6decbd0 2024-04-22T01:12:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/504847125 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9abc04b5e0 con 0x560ea6decbd0 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 10 client.15447 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 10 client.15447 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 20 client.15447 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6dfe1f0 0x560ea6dff660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 4 client.15447 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.976+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ea6d59330 con 0x560ea6dfe1f0 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.977+0000 7f9ac6097640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6dfe1f0 0x560ea6dff660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.977+0000 7f9aa9ffb640 10 client.15447 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.977+0000 7f9ac6097640 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6dfe1f0 0x560ea6dff660 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1ed5b8604755e25 server_cookie=8a627ef867f9a in_seq=0 out_seq=0 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/504847125 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ea6d59330 con 0x560ea6dfe1f0 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9aa9ffb640 10 client.15447 handle_command_reply: tid=0 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 client.15447 shutdown 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 msgr2=0x560ea6df8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 0x560ea6df8a80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9abc02c7b0 tx=0x7f9abc005d50 comp rx=0 tx=0).stop 2024-04-22T01:12:56.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 shutdown_connections 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6dfe1f0 0x560ea6dff660 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6decbd0 0x560ea6df8a80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6df1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9aa9ffb640 0 client.15447 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 --2- 172.21.15.38:0/504847125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7070 0x560ea6df1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 >> 172.21.15.38:0/504847125 conn(0x560ea6de25e0 msgr2=0x560ea6de1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 shutdown_connections 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 1 -- 172.21.15.38:0/504847125 wait complete. 2024-04-22T01:12:56.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.978+0000 7f9acafc1740 20 client.15447 trim_cache size 0 max 16384 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 Processor -- start 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 -- start start 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6de9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6decbd0 0x560ea6df8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ea6dfd9a0 con 0x560ea6de7070 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560ea6dfe070 con 0x560ea6decbd0 2024-04-22T01:12:57.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9acafc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560ea6dfe740 con 0x560ea6de7a60 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50900/0 (socket says 172.21.15.38:50900) 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 -- 172.21.15.38:0/24114703 learned_addr learned my addr 172.21.15.38:0/24114703 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6898640 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6decbd0 0x560ea6df8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 -- 172.21.15.38:0/24114703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 msgr2=0x560ea6de9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6de9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 -- 172.21.15.38:0/24114703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6decbd0 msgr2=0x560ea6df8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6decbd0 0x560ea6df8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 -- 172.21.15.38:0/24114703 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea6de7e40 con 0x560ea6de7070 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.997+0000 7f9ac6097640 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de8f30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9ab809bfd0 tx=0x7f9ab801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab63230f45bff67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9aabfff640 1 -- 172.21.15.38:0/24114703 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80a03f0 con 0x560ea6de7070 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9aabfff640 1 -- 172.21.15.38:0/24114703 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab80b4030 con 0x560ea6de7070 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9aabfff640 1 -- 172.21.15.38:0/24114703 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80202b0 con 0x560ea6de7070 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 -- 172.21.15.38:0/24114703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 msgr2=0x560ea6de8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de8f30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9ab809bfd0 tx=0x7f9ab801c850 comp rx=0 tx=0).stop 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 -- 172.21.15.38:0/24114703 shutdown_connections 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6decbd0 0x560ea6df8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6de7a60 0x560ea6de9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 --2- 172.21.15.38:0/24114703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6de8f30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 -- 172.21.15.38:0/24114703 >> 172.21.15.38:0/24114703 conn(0x560ea6de25e0 msgr2=0x560ea6df1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 -- 172.21.15.38:0/24114703 shutdown_connections 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.998+0000 7f9acafc1740 1 -- 172.21.15.38:0/24114703 wait complete. 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 -- start start 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 Processor -- start 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6df8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6df52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6df5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560ea6dff4e0 con 0x560ea6de7070 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560ea6dfee10 con 0x560ea6de7a60 2024-04-22T01:12:57.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560ea6dffbb0 con 0x560ea6decbd0 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9acafc1740 10 client.0 _ll_register_callbacks cb 0x7f9abc005580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6df8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6df8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50916/0 (socket says 172.21.15.38:50916) 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 -- 172.21.15.38:0/851092011 learned_addr learned my addr 172.21.15.38:0/851092011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac5896640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6df52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6898640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6df5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 -- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 msgr2=0x560ea6df5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6df5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 -- 172.21.15.38:0/851092011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 msgr2=0x560ea6df52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6df52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 -- 172.21.15.38:0/851092011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea6df5ea0 con 0x560ea6de7070 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac5896640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6df52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6898640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6df5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:57.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9ac6097640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6df8820 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8007fd0 tx=0x7f9ab809bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23311f364f632fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:56.999+0000 7f9aa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/851092011 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8095040 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/851092011 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9ab809a0e0 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ab80b4030 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/851092011 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9ab809a340 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 10 client.15453 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560ea6e652a0 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/851092011 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab80b4030 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/851092011 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9ab800f040 con 0x560ea6de7070 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 10 client.15453 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 10 client.15453 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 20 client.15453 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6e69920 0x560ea6e66430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 4 client.15453 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:57.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "status"}) -- 0x560ea6d53680 con 0x560ea6e69920 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.000+0000 7f9ac5896640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6e69920 0x560ea6e66430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9ac5896640 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6e69920 0x560ea6e66430 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=74df86c82a99e02 server_cookie=9d9640ea70a93079 in_seq=0 out_seq=0 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9aa9ffb640 10 client.15453 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9aa9ffb640 1 -- 172.21.15.38:0/851092011 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560ea6d53680 con 0x560ea6e69920 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9aa9ffb640 10 client.15453 handle_command_reply: tid=0 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 client.15453 shutdown 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 msgr2=0x560ea6df8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6df8820 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8007fd0 tx=0x7f9ab809bc00 comp rx=0 tx=0).stop 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 shutdown_connections 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x560ea6e69920 0x560ea6e66430 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea6decbd0 0x560ea6df5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ea6de7a60 0x560ea6df52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9aa9ffb640 0 client.15453 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 --2- 172.21.15.38:0/851092011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560ea6de7070 0x560ea6df8820 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 >> 172.21.15.38:0/851092011 conn(0x560ea6de25e0 msgr2=0x560ea6df14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 shutdown_connections 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 1 -- 172.21.15.38:0/851092011 wait complete. 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.001+0000 7f9acafc1740 20 client.15453 trim_cache size 0 max 16384 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24472, 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:57.008 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:57.009 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 119.10659616300001, 2024-04-22T01:12:57.009 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:57.009 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:57.009 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:57.009 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 159.22601741099999 2024-04-22T01:12:57.009 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9aac0a2c10 msgr2=0x7f9aac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9aac0a2c10 0x7f9aac0a50d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9abc036040 tx=0x7f9abc009260 comp rx=0 tx=0).stop 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 msgr2=0x7f9ac0101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 0x7f9ac0101b70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800eab0 tx=0x7f9ab800ef70 comp rx=0 tx=0).stop 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 shutdown_connections 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9aac0a2c10 0x7f9aac0a50d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ac0107460 0x7f9ac01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ac0104b30 0x7f9ac01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 --2- 172.21.15.38:0/504198720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ac0102200 0x7f9ac0101b70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 >> 172.21.15.38:0/504198720 conn(0x7f9ac00fbbf0 msgr2=0x7f9ac0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 shutdown_connections 2024-04-22T01:12:57.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.002+0000 7f9ac82fb640 1 -- 172.21.15.38:0/504198720 wait complete. 2024-04-22T01:12:57.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T01:12:57.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3589997500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:57.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/698316082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:57.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3174552698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 -- 172.21.15.38:0/2244631777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 msgr2=0x7fea34109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/2244631777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34109be0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009ca0 tx=0x7fea2c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 -- 172.21.15.38:0/2244631777 shutdown_connections 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/2244631777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34109be0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/2244631777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 0x7fea34102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/2244631777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea341017a0 0x7fea34101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 -- 172.21.15.38:0/2244631777 >> 172.21.15.38:0/2244631777 conn(0x7fea340fcd10 msgr2=0x7fea340ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 -- 172.21.15.38:0/2244631777 shutdown_connections 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.115+0000 7fea3a8a7640 1 -- 172.21.15.38:0/2244631777 wait complete. 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 Processor -- start 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 -- start start 2024-04-22T01:12:57.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea341017a0 0x7fea340754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 0x7fea34073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea3410d420 con 0x7fea34103350 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fea3410d2a0 con 0x7fea341017a0 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea3a8a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fea3410d5a0 con 0x7fea34102990 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea38e44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea38e44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50926/0 (socket says 172.21.15.38:50926) 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.116+0000 7fea38e44640 1 -- 172.21.15.38:0/3822644855 learned_addr learned my addr 172.21.15.38:0/3822644855 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea33fff640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea341017a0 0x7fea340754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea337fe640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 0x7fea34073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea38e44640 1 -- 172.21.15.38:0/3822644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 msgr2=0x7fea34073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea38e44640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 0x7fea34073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea38e44640 1 -- 172.21.15.38:0/3822644855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea341017a0 msgr2=0x7fea340754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea38e44640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea341017a0 0x7fea340754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea38e44640 1 -- 172.21.15.38:0/3822644855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea34074900 con 0x7fea34103350 2024-04-22T01:12:57.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea337fe640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 0x7fea34073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:57.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea38e44640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34074070 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fea2c037040 tx=0x7fea2c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea317fa640 1 -- 172.21.15.38:0/3822644855 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c041020 con 0x7fea34103350 2024-04-22T01:12:57.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea317fa640 1 -- 172.21.15.38:0/3822644855 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea2c034d90 con 0x7fea34103350 2024-04-22T01:12:57.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea341a1b20 con 0x7fea34103350 2024-04-22T01:12:57.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea317fa640 1 -- 172.21.15.38:0/3822644855 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c039040 con 0x7fea34103350 2024-04-22T01:12:57.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.117+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea341a2070 con 0x7fea34103350 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.119+0000 7fea317fa640 1 -- 172.21.15.38:0/3822644855 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fea2c044020 con 0x7fea34103350 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 Processor -- start 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 -- start start 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa883490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa888560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa7e7df0 con 0x55d0fa883070 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa790640 con 0x55d0fa883a60 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea3d56e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa7e3db0 con 0x55d0fa888bd0 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea317fa640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fea1c0a2b90 0x7fea1c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa883490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa883490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50932/0 (socket says 172.21.15.38:50932) 2024-04-22T01:12:57.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 -- 172.21.15.38:0/357947200 learned_addr learned my addr 172.21.15.38:0/357947200 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea38e44640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fea1c0a2b90 0x7fea1c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea317fa640 1 -- 172.21.15.38:0/3822644855 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fea2c0f13e0 con 0x7fea34103350 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea33fff640 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa888560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 -- 172.21.15.38:0/357947200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 msgr2=0x55d0fa88b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 -- 172.21.15.38:0/357947200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 msgr2=0x55d0fa888560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.120+0000 7fea337fe640 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa888560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea337fe640 1 -- 172.21.15.38:0/357947200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fa4fc2d0 con 0x55d0fa883070 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea33fff640 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa888560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea337fe640 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa883490 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fea24013590 tx=0x7fea240097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28422cf38060961 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea38e44640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fea1c0a2b90 0x7fea1c0a5050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0bd7c0 tx=0x7fea2c08d6e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea1a7fc640 1 -- 172.21.15.38:0/357947200 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2400f520 con 0x55d0fa883070 2024-04-22T01:12:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea1a7fc640 1 -- 172.21.15.38:0/357947200 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea2400fba0 con 0x55d0fa883070 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea1a7fc640 1 -- 172.21.15.38:0/357947200 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea240109c0 con 0x55d0fa883070 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea3d56e740 1 -- 172.21.15.38:0/357947200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 msgr2=0x55d0fa883490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.121+0000 7fea3d56e740 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa883490 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fea24013590 tx=0x7fea240097d0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- 172.21.15.38:0/357947200 shutdown_connections 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa888560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 --2- 172.21.15.38:0/357947200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa883490 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- 172.21.15.38:0/357947200 >> 172.21.15.38:0/357947200 conn(0x55d0fa87e5e0 msgr2=0x55d0fa87e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- 172.21.15.38:0/357947200 shutdown_connections 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- 172.21.15.38:0/357947200 wait complete. 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- start start 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 Processor -- start 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa88daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa894ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa88bb80 con 0x55d0fa883070 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa7a3be0 con 0x55d0fa883a60 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa88bd00 con 0x55d0fa888bd0 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea3d56e740 10 client.0 _ll_register_callbacks cb 0x55d0fa7a35d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50948/0 (socket says 172.21.15.38:50948) 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.122+0000 7fea337fe640 1 -- 172.21.15.38:0/3879075922 learned_addr learned my addr 172.21.15.38:0/3879075922 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea337fe640 1 -- 172.21.15.38:0/3879075922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 msgr2=0x55d0fa894ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea337fe640 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa894ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea337fe640 1 -- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 msgr2=0x55d0fa88daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea33fff640 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa88daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea337fe640 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa88daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea337fe640 1 -- 172.21.15.38:0/3879075922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fa5c66b0 con 0x55d0fa883070 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea337fe640 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88d560 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fea24013ee0 tx=0x7fea2400ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efc7cb808d09366 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea137fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea137fe640 1 -- 172.21.15.38:0/3879075922 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24009900 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea137fe640 1 -- 172.21.15.38:0/3879075922 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fea24005260 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea2400fba0 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea137fe640 1 -- 172.21.15.38:0/3879075922 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea24032050 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea3d56e740 10 client.15465 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0fa897ac0 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.123+0000 7fea137fe640 1 -- 172.21.15.38:0/3879075922 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea2400fba0 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea137fe640 1 -- 172.21.15.38:0/3879075922 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fea240046c0 con 0x55d0fa883070 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea3d56e740 10 client.15465 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea3d56e740 10 client.15465 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea3d56e740 20 client.15465 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea3d56e740 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa899fe0 0x55d0fa89b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea3d56e740 4 client.15465 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0fa7f5330 con 0x55d0fa899fe0 2024-04-22T01:12:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea33fff640 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa899fe0 0x55d0fa89b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea137fe640 10 client.15465 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.124+0000 7fea33fff640 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa899fe0 0x55d0fa89b450 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72d1b84b81efa658 server_cookie=f45aee58aa43fbdd in_seq=0 out_seq=0 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.125+0000 7fea137fe640 1 -- 172.21.15.38:0/3879075922 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0fa7f5330 con 0x55d0fa899fe0 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.125+0000 7fea137fe640 10 client.15465 handle_command_reply: tid=0 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.125+0000 7fea3d56e740 1 client.15465 shutdown 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.125+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 msgr2=0x55d0fa88d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.125+0000 7fea3d56e740 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88d560 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fea24013ee0 tx=0x7fea2400ffd0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 shutdown_connections 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa899fe0 0x55d0fa89b450 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa894ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa88daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea137fe640 0 client.15465 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 --2- 172.21.15.38:0/3879075922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88d560 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 >> 172.21.15.38:0/3879075922 conn(0x55d0fa87e5e0 msgr2=0x55d0fa87d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 shutdown_connections 2024-04-22T01:12:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 1 -- 172.21.15.38:0/3879075922 wait complete. 2024-04-22T01:12:57.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.126+0000 7fea3d56e740 20 client.15465 trim_cache size 0 max 16384 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 Processor -- start 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 -- start start 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa894a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa899790 con 0x55d0fa883070 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa899e60 con 0x55d0fa883a60 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea3d56e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa89a530 con 0x55d0fa888bd0 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50950/0 (socket says 172.21.15.38:50950) 2024-04-22T01:12:57.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 -- 172.21.15.38:0/1277730474 learned_addr learned my addr 172.21.15.38:0/1277730474 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 -- 172.21.15.38:0/1277730474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 msgr2=0x55d0fa88d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 -- 172.21.15.38:0/1277730474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 msgr2=0x55d0fa894a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa894a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 -- 172.21.15.38:0/1277730474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fa88e550 con 0x55d0fa883070 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.144+0000 7fea337fe640 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88c130 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fea2400f8b0 tx=0x7fea24009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452d00fe068590fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea19ffb640 1 -- 172.21.15.38:0/1277730474 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea240130d0 con 0x55d0fa883070 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea19ffb640 1 -- 172.21.15.38:0/1277730474 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea24026030 con 0x55d0fa883070 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea19ffb640 1 -- 172.21.15.38:0/1277730474 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24011470 con 0x55d0fa883070 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 -- 172.21.15.38:0/1277730474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 msgr2=0x55d0fa88c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88c130 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fea2400f8b0 tx=0x7fea24009a00 comp rx=0 tx=0).stop 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 -- 172.21.15.38:0/1277730474 shutdown_connections 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa888bd0 0x55d0fa88d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa883a60 0x55d0fa894a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 --2- 172.21.15.38:0/1277730474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa88c130 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 -- 172.21.15.38:0/1277730474 >> 172.21.15.38:0/1277730474 conn(0x55d0fa87e5e0 msgr2=0x55d0fa87e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 -- 172.21.15.38:0/1277730474 shutdown_connections 2024-04-22T01:12:57.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.145+0000 7fea3d56e740 1 -- 172.21.15.38:0/1277730474 wait complete. 2024-04-22T01:12:57.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 -- start start 2024-04-22T01:12:57.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 Processor -- start 2024-04-22T01:12:57.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3589997500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:12:57.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/698316082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:12:57.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3174552698' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa890ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa883a60 0x55d0fa890ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa888bd0 0x55d0fa894010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa89b2d0 con 0x55d0fa883070 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa89ac00 con 0x55d0fa888bd0 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0fa89b9a0 con 0x55d0fa883a60 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea3d56e740 10 client.0 _ll_register_callbacks cb 0x7fea2400f970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa890ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa890ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50952/0 (socket says 172.21.15.38:50952) 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 -- 172.21.15.38:0/4142773636 learned_addr learned my addr 172.21.15.38:0/4142773636 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 -- 172.21.15.38:0/4142773636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa883a60 msgr2=0x55d0fa890ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea33fff640 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa883a60 0x55d0fa890ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa883a60 0x55d0fa890ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 -- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa888bd0 msgr2=0x55d0fa894010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa888bd0 0x55d0fa894010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.146+0000 7fea337fe640 1 -- 172.21.15.38:0/4142773636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fa894600 con 0x55d0fa883070 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea337fe640 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa890ab0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fea24004c80 tx=0x7fea24013160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b328548b6cee97c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea137fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea137fe640 1 -- 172.21.15.38:0/4142773636 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24004d00 con 0x55d0fa883070 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea24026030 con 0x55d0fa883070 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea137fe640 1 -- 172.21.15.38:0/4142773636 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fea24004660 con 0x55d0fa883070 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea137fe640 1 -- 172.21.15.38:0/4142773636 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea24004ea0 con 0x55d0fa883070 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 10 client.15471 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0fa9008f0 con 0x55d0fa883070 2024-04-22T01:12:57.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea137fe640 1 -- 172.21.15.38:0/4142773636 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea24026030 con 0x55d0fa883070 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea137fe640 1 -- 172.21.15.38:0/4142773636 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fea24011900 con 0x55d0fa883070 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 10 client.15471 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 10 client.15471 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 20 client.15471 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa905100 0x55d0fa903e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 4 client.15471 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.147+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "status"}) -- 0x55d0fa7ef680 con 0x55d0fa905100 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.148+0000 7fea33fff640 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa905100 0x55d0fa903e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.148+0000 7fea137fe640 10 client.15471 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.148+0000 7fea33fff640 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa905100 0x55d0fa903e10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=921daf366d3feaad server_cookie=97100c6a61007cc8 in_seq=0 out_seq=0 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.148+0000 7fea137fe640 1 -- 172.21.15.38:0/4142773636 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55d0fa7ef680 con 0x55d0fa905100 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.148+0000 7fea137fe640 10 client.15471 handle_command_reply: tid=0 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.148+0000 7fea3d56e740 1 client.15471 shutdown 2024-04-22T01:12:57.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 msgr2=0x55d0fa890ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa890ab0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fea24004c80 tx=0x7fea24013160 comp rx=0 tx=0).stop 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 shutdown_connections 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55d0fa905100 0x55d0fa903e10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0fa888bd0 0x55d0fa894010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fa883a60 0x55d0fa890ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea137fe640 0 client.15471 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 --2- 172.21.15.38:0/4142773636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0fa883070 0x55d0fa890ab0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 >> 172.21.15.38:0/4142773636 conn(0x55d0fa87e5e0 msgr2=0x55d0fa88d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 shutdown_connections 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 1 -- 172.21.15.38:0/4142773636 wait complete. 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3d56e740 20 client.15471 trim_cache size 0 max 16384 2024-04-22T01:12:57.155 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24485, 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 120.269440973, 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 162.362181714 2024-04-22T01:12:57.156 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fea1c0a2b90 msgr2=0x7fea1c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fea1c0a2b90 0x7fea1c0a5050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0bd7c0 tx=0x7fea2c08d6e0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 msgr2=0x7fea34074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.149+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34074070 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fea2c037040 tx=0x7fea2c032f20 comp rx=0 tx=0).stop 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 shutdown_connections 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fea1c0a2b90 0x7fea1c0a5050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea34103350 0x7fea34074070 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea34102990 0x7fea34073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 --2- 172.21.15.38:0/3822644855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea341017a0 0x7fea340754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 >> 172.21.15.38:0/3822644855 conn(0x7fea340fcd10 msgr2=0x7fea34105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 shutdown_connections 2024-04-22T01:12:57.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.150+0000 7fea3a8a7640 1 -- 172.21.15.38:0/3822644855 wait complete. 2024-04-22T01:12:57.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.263+0000 7f06efdde640 1 -- 172.21.15.38:0/451572843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8101870 msgr2=0x7f06e8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.263+0000 7f06efdde640 1 --2- 172.21.15.38:0/451572843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8101870 0x7f06e8101cf0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009900 tx=0x7f06d802bc30 comp rx=0 tx=0).stop 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- 172.21.15.38:0/451572843 shutdown_connections 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 --2- 172.21.15.38:0/451572843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e8102230 0x7f06e8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 --2- 172.21.15.38:0/451572843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8101870 0x7f06e8101cf0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 --2- 172.21.15.38:0/451572843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e8100680 0x7f06e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- 172.21.15.38:0/451572843 >> 172.21.15.38:0/451572843 conn(0x7f06e80fbbf0 msgr2=0x7f06e80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- 172.21.15.38:0/451572843 shutdown_connections 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- 172.21.15.38:0/451572843 wait complete. 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 Processor -- start 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- start start 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 0x7f06e8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e8101870 0x7f06e8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e8102230 0x7f06e819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06e810c360 con 0x7f06e8100680 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06e810c1e0 con 0x7f06e8101870 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.264+0000 7f06efdde640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06e810c4e0 con 0x7f06e8102230 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 0x7f06e8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 0x7f06e8197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50968/0 (socket says 172.21.15.38:50968) 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 -- 172.21.15.38:0/2138700918 learned_addr learned my addr 172.21.15.38:0/2138700918 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06ed379640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e8101870 0x7f06e8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 -- 172.21.15.38:0/2138700918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e8102230 msgr2=0x7f06e819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06ee37b640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e8102230 0x7f06e819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e8102230 0x7f06e819f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 -- 172.21.15.38:0/2138700918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e8101870 msgr2=0x7f06e8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e8101870 0x7f06e8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 -- 172.21.15.38:0/2138700918 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e819f770 con 0x7f06e8100680 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06edb7a640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 0x7f06e8197b50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c970 tx=0x7f06e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06deffd640 1 -- 172.21.15.38:0/2138700918 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4004220 con 0x7f06e8100680 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e819fa60 con 0x7f06e8100680 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06deffd640 1 -- 172.21.15.38:0/2138700918 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e40043c0 con 0x7f06e8100680 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06deffd640 1 -- 172.21.15.38:0/2138700918 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e400f450 con 0x7f06e8100680 2024-04-22T01:12:57.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.265+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e819ff80 con 0x7f06e8100680 2024-04-22T01:12:57.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.268+0000 7f06deffd640 1 -- 172.21.15.38:0/2138700918 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f06e40079b0 con 0x7f06e8100680 2024-04-22T01:12:57.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.268+0000 7f06f29c8740 1 Processor -- start 2024-04-22T01:12:57.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 -- start start 2024-04-22T01:12:57.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403070 0x55c8aa403490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 0x55c8aa408560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa40b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa367df0 con 0x55c8aa403a60 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa310640 con 0x55c8aa403070 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06f29c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa363db0 con 0x55c8aa408bd0 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 0x55c8aa408560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 0x55c8aa408560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50974/0 (socket says 172.21.15.38:50974) 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 -- 172.21.15.38:0/218294344 learned_addr learned my addr 172.21.15.38:0/218294344 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06deffd640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06d40a2b90 0x7f06d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06ed379640 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa40b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 -- 172.21.15.38:0/218294344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 msgr2=0x55c8aa40b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa40b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06ee37b640 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403070 0x55c8aa403490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 -- 172.21.15.38:0/218294344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403070 msgr2=0x55c8aa403490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403070 0x55c8aa403490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06edb7a640 1 -- 172.21.15.38:0/218294344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8aa07c2d0 con 0x55c8aa403a60 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06ee37b640 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403070 0x55c8aa403490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06ee37b640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06d40a2b90 0x7f06d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.269+0000 7f06deffd640 1 -- 172.21.15.38:0/2138700918 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f06e40cf250 con 0x7f06e8100680 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.270+0000 7f06edb7a640 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 0x55c8aa408560 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f06e405a030 tx=0x7f06e40978e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e80aa89087fb77 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.270+0000 7f06ee37b640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06d40a2b90 0x7f06d40a5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55c8aa4042c0 tx=0x7f06e0004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.270+0000 7f06d3fff640 1 -- 172.21.15.38:0/218294344 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e401d030 con 0x55c8aa403a60 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.270+0000 7f06d3fff640 1 -- 172.21.15.38:0/218294344 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e409bde0 con 0x55c8aa403a60 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.270+0000 7f06d3fff640 1 -- 172.21.15.38:0/218294344 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e405ae60 con 0x55c8aa403a60 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- 172.21.15.38:0/218294344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 msgr2=0x55c8aa408560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 0x55c8aa408560 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f06e405a030 tx=0x7f06e40978e0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- 172.21.15.38:0/218294344 shutdown_connections 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa40b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403a60 0x55c8aa408560 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- 172.21.15.38:0/218294344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403070 0x55c8aa403490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- 172.21.15.38:0/218294344 >> 172.21.15.38:0/218294344 conn(0x55c8aa3fe5e0 msgr2=0x55c8aa3fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- 172.21.15.38:0/218294344 shutdown_connections 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- 172.21.15.38:0/218294344 wait complete. 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- start start 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 Processor -- start 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403a60 0x55c8aa40d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa408bd0 0x55c8aa414a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa40bb20 con 0x55c8aa403070 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa323be0 con 0x55c8aa408bd0 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.271+0000 7f06f29c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa40bca0 con 0x55c8aa403a60 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06f29c8740 10 client.0 _ll_register_callbacks cb 0x55c8aa3235d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50978/0 (socket says 172.21.15.38:50978) 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 -- 172.21.15.38:0/1719439910 learned_addr learned my addr 172.21.15.38:0/1719439910 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06ee37b640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa408bd0 0x55c8aa414a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06ed379640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403a60 0x55c8aa40d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 -- 172.21.15.38:0/1719439910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403a60 msgr2=0x55c8aa40d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403a60 0x55c8aa40d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 -- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa408bd0 msgr2=0x55c8aa414a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa408bd0 0x55c8aa414a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 -- 172.21.15.38:0/1719439910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8aa1466b0 con 0x55c8aa403070 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06ee37b640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa408bd0 0x55c8aa414a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:57.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06ed379640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403a60 0x55c8aa40d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06edb7a640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d490 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c940 tx=0x7f06e409b0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214a9e8bceb3dc45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06d17fa640 1 -- 172.21.15.38:0/1719439910 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e401d040 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06d17fa640 1 -- 172.21.15.38:0/1719439910 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f06e4003610 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06e409bde0 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06d17fa640 1 -- 172.21.15.38:0/1719439910 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f06e40a4350 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06f29c8740 10 client.15483 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c8aa417960 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.272+0000 7f06d17fa640 1 -- 172.21.15.38:0/1719439910 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e409bde0 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06d17fa640 1 -- 172.21.15.38:0/1719439910 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f06e409abc0 con 0x55c8aa403070 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06f29c8740 10 client.15483 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06f29c8740 10 client.15483 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06f29c8740 20 client.15483 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa419bd0 0x55c8aa41b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06f29c8740 4 client.15483 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8aa375330 con 0x55c8aa419bd0 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06ed379640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa419bd0 0x55c8aa41b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06d17fa640 10 client.15483 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.273+0000 7f06ed379640 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa419bd0 0x55c8aa41b040 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7847f58597ae91c server_cookie=d2753d20d42ef6e8 in_seq=0 out_seq=0 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06d17fa640 1 -- 172.21.15.38:0/1719439910 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8aa375330 con 0x55c8aa419bd0 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06d17fa640 10 client.15483 handle_command_reply: tid=0 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 client.15483 shutdown 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 msgr2=0x55c8aa40d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d490 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c940 tx=0x7f06e409b0f0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 shutdown_connections 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa419bd0 0x55c8aa41b040 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa408bd0 0x55c8aa414a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06d17fa640 0 client.15483 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403a60 0x55c8aa40d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1719439910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d490 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 >> 172.21.15.38:0/1719439910 conn(0x55c8aa3fe5e0 msgr2=0x55c8aa3fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 shutdown_connections 2024-04-22T01:12:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.274+0000 7f06f29c8740 1 -- 172.21.15.38:0/1719439910 wait complete. 2024-04-22T01:12:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.275+0000 7f06f29c8740 20 client.15483 trim_cache size 0 max 16384 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 Processor -- start 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 -- start start 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4149a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa418cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa419380 con 0x55c8aa403070 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa419a50 con 0x55c8aa403a60 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06f29c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa41a120 con 0x55c8aa408bd0 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50994/0 (socket says 172.21.15.38:50994) 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 -- 172.21.15.38:0/1983701450 learned_addr learned my addr 172.21.15.38:0/1983701450 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06ed379640 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4149a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 -- 172.21.15.38:0/1983701450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 msgr2=0x55c8aa418cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa418cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 -- 172.21.15.38:0/1983701450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 msgr2=0x55c8aa4149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4149a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.293+0000 7f06edb7a640 1 -- 172.21.15.38:0/1983701450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8aa403e40 con 0x55c8aa403070 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06edb7a640 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d930 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f06e4097b40 tx=0x7f06e409fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e4748251aa876f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06d37fe640 1 -- 172.21.15.38:0/1983701450 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e401d040 con 0x55c8aa403070 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06d37fe640 1 -- 172.21.15.38:0/1983701450 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e409bde0 con 0x55c8aa403070 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06d37fe640 1 -- 172.21.15.38:0/1983701450 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4007e50 con 0x55c8aa403070 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 -- 172.21.15.38:0/1983701450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 msgr2=0x55c8aa40d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d930 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f06e4097b40 tx=0x7f06e409fc10 comp rx=0 tx=0).stop 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 -- 172.21.15.38:0/1983701450 shutdown_connections 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa408bd0 0x55c8aa418cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 --2- 172.21.15.38:0/1983701450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa403070 0x55c8aa40d930 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 -- 172.21.15.38:0/1983701450 >> 172.21.15.38:0/1983701450 conn(0x55c8aa3fe5e0 msgr2=0x55c8aa4084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 -- 172.21.15.38:0/1983701450 shutdown_connections 2024-04-22T01:12:57.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.294+0000 7f06f29c8740 1 -- 172.21.15.38:0/1983701450 wait complete. 2024-04-22T01:12:57.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 -- start start 2024-04-22T01:12:57.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 Processor -- start 2024-04-22T01:12:57.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403070 0x55c8aa410b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4110e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 0x55c8aa413d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa41aec0 con 0x55c8aa408bd0 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa41a7f0 con 0x55c8aa403a60 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c8aa41b590 con 0x55c8aa403070 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06f29c8740 10 client.0 _ll_register_callbacks cb 0x7f06e40076c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 0x55c8aa413d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 0x55c8aa413d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51008/0 (socket says 172.21.15.38:51008) 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 -- 172.21.15.38:0/2097591904 learned_addr learned my addr 172.21.15.38:0/2097591904 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ed379640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06edb7a640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403070 0x55c8aa410b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 -- 172.21.15.38:0/2097591904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403070 msgr2=0x55c8aa410b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403070 0x55c8aa410b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 -- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 msgr2=0x55c8aa4110e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4110e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.295+0000 7f06ee37b640 1 -- 172.21.15.38:0/2097591904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8aa414320 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06ed379640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06ee37b640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 0x55c8aa413d30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f06e0006310 tx=0x7f06e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113dc0ab9571c769 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06d17fa640 1 -- 172.21.15.38:0/2097591904 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e0027060 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06d17fa640 1 -- 172.21.15.38:0/2097591904 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f06e0012920 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06e409bde0 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06d17fa640 1 -- 172.21.15.38:0/2097591904 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f06e002e050 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 10 client.15489 fetch_fsmap learned FSMap version 27 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c8aa480350 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06d17fa640 1 -- 172.21.15.38:0/2097591904 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e409bde0 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06d17fa640 1 -- 172.21.15.38:0/2097591904 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f06e0013ba0 con 0x55c8aa408bd0 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 10 client.15489 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T01:12:57.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 10 client.15489 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 20 client.15489 populate_metadata read hostname 'smithi038' 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa484ea0 0x55c8aa483bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 4 client.15489 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.296+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "status"}) -- 0x55c8aa36f680 con 0x55c8aa484ea0 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06edb7a640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa484ea0 0x55c8aa483bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06d17fa640 10 client.15489 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06edb7a640 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa484ea0 0x55c8aa483bb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3c9009a336cb64d server_cookie=3fa64ae1fe640fea in_seq=0 out_seq=0 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06d17fa640 1 -- 172.21.15.38:0/2097591904 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c8aa36f680 con 0x55c8aa484ea0 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06d17fa640 10 client.15489 handle_command_reply: tid=0 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 client.15489 shutdown 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 msgr2=0x55c8aa413d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 0x55c8aa413d30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f06e0006310 tx=0x7f06e0004a40 comp rx=0 tx=0).stop 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 shutdown_connections 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c8aa484ea0 0x55c8aa483bb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8aa408bd0 0x55c8aa413d30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c8aa403a60 0x55c8aa4110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06f29c8740 1 --2- 172.21.15.38:0/2097591904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c8aa403070 0x55c8aa410b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.297+0000 7f06d17fa640 0 client.15489 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 >> 172.21.15.38:0/2097591904 conn(0x55c8aa3fe5e0 msgr2=0x55c8aa3f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 shutdown_connections 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06f29c8740 1 -- 172.21.15.38:0/2097591904 wait complete. 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06f29c8740 20 client.15489 trim_cache size 0 max 16384 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "caf31b2e-0043-11ef-bc93-c7b262605968", 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-04-22T01:12:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24527, 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 146.704933283, 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 93, 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 150.86422074199999 2024-04-22T01:12:57.305 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06d40a2b90 msgr2=0x7f06d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06d40a2b90 0x7f06d40a5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55c8aa4042c0 tx=0x7f06e0004b50 comp rx=0 tx=0).stop 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 msgr2=0x7f06e8197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 0x7f06e8197b50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c970 tx=0x7f06e400ce30 comp rx=0 tx=0).stop 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 shutdown_connections 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06d40a2b90 0x7f06d40a5050 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e8102230 0x7f06e819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e8101870 0x7f06e8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 --2- 172.21.15.38:0/2138700918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06e8100680 0x7f06e8197b50 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:57.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 >> 172.21.15.38:0/2138700918 conn(0x7f06e80fbbf0 msgr2=0x7f06e80fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:57.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 shutdown_connections 2024-04-22T01:12:57.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:57.298+0000 7f06efdde640 1 -- 172.21.15.38:0/2138700918 wait complete. 2024-04-22T01:12:57.309 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T01:12:57.320 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T01:12:57.320 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T01:12:57.321 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T01:12:57.321 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T01:12:57.332 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T01:12:57.332 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-22T01:12:57.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:12:57.332 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T01:12:57.348 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-22T01:12:57.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:12:57.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T01:12:57.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:12:57.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-04-22T01:12:57.509 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-22T01:12:57.510 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T01:12:57.510 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:12:57.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T01:12:57.759 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:12:58.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:57 smithi203 ceph-mon[31825]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:58.016 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:57 smithi038 ceph-mon[28043]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:58.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:57 smithi111 ceph-mon[31057]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:12:59.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 -- 172.21.15.38:0/344412095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98241071a0 msgr2=0x7f9824107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:59.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 --2- 172.21.15.38:0/344412095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98241071a0 0x7f9824107580 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9814009990 tx=0x7f9814009ca0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 -- 172.21.15.38:0/344412095 shutdown_connections 2024-04-22T01:12:59.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 --2- 172.21.15.38:0/344412095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9824101af0 0x7f982410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 --2- 172.21.15.38:0/344412095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98241071a0 0x7f9824107580 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 --2- 172.21.15.38:0/344412095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9824101130 0x7f98241015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 -- 172.21.15.38:0/344412095 >> 172.21.15.38:0/344412095 conn(0x7f98240780e0 msgr2=0x7f98240784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.485+0000 7f982a8c0700 1 -- 172.21.15.38:0/344412095 shutdown_connections 2024-04-22T01:12:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.486+0000 7f982a8c0700 1 -- 172.21.15.38:0/344412095 wait complete. 2024-04-22T01:12:59.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 Processor -- start 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 -- start start 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 0x7f98241998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9824101af0 0x7f9824199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98241071a0 0x7f98241a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9824111410 con 0x7f9824101130 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9824111290 con 0x7f9824101af0 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f982a8c0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9824111590 con 0x7f98241071a0 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f9823fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 0x7f98241998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f9823fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 0x7f98241998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51030/0 (socket says 172.21.15.38:51030) 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.487+0000 7f9823fff700 1 -- 172.21.15.38:0/1023725778 learned_addr learned my addr 172.21.15.38:0/1023725778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f98237fe700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9824101af0 0x7f9824199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9828e5d700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98241071a0 0x7f98241a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9823fff700 1 -- 172.21.15.38:0/1023725778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98241071a0 msgr2=0x7f98241a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9823fff700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98241071a0 0x7f98241a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9823fff700 1 -- 172.21.15.38:0/1023725778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9824101af0 msgr2=0x7f9824199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9823fff700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9824101af0 0x7f9824199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9823fff700 1 -- 172.21.15.38:0/1023725778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98241976c0 con 0x7f9824101130 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f9823fff700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 0x7f98241998d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9814000c00 tx=0x7f98140047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814027070 con 0x7f9824101130 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9814004050 con 0x7f9824101130 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981400f460 con 0x7f9824101130 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9824197950 con 0x7f9824101130 2024-04-22T01:12:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.488+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9824197de0 con 0x7f9824101130 2024-04-22T01:12:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.489+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9824068f20 con 0x7f9824101130 2024-04-22T01:12:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.490+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f981402a070 con 0x7f9824101130 2024-04-22T01:12:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.491+0000 7f98217fa700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f980c0a2ca0 0x7f980c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:12:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.491+0000 7f98237fe700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f980c0a2ca0 0x7f980c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:12:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.491+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f98140d0300 con 0x7f9824101130 2024-04-22T01:12:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.492+0000 7f98237fe700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f980c0a2ca0 0x7f980c0a5160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9818005950 tx=0x7f98180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:12:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.496+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f981405e6f0 con 0x7f9824101130 2024-04-22T01:12:59.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.748+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f982402db30 con 0x7f9824101130 2024-04-22T01:12:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:12:59 smithi038 ceph-mon[28043]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:12:59.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.749+0000 7f98217fa700 1 -- 172.21.15.38:0/1023725778 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f9814013070 con 0x7f9824101130 2024-04-22T01:12:59.750 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T01:12:59.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f980c0a2ca0 msgr2=0x7f980c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f980c0a2ca0 0x7f980c0a5160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9818005950 tx=0x7f98180058e0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 msgr2=0x7f98241998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 0x7f98241998d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9814000c00 tx=0x7f98140047b0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 shutdown_connections 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f980c0a2ca0 0x7f980c0a5160 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98241071a0 0x7f98241a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9824101130 0x7f98241998d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 --2- 172.21.15.38:0/1023725778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9824101af0 0x7f9824199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 >> 172.21.15.38:0/1023725778 conn(0x7f98240780e0 msgr2=0x7f98240ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 shutdown_connections 2024-04-22T01:12:59.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:12:59.751+0000 7f982a8c0700 1 -- 172.21.15.38:0/1023725778 wait complete. 2024-04-22T01:13:00.018 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:12:59 smithi203 ceph-mon[31825]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:13:00.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:12:59 smithi111 ceph-mon[31057]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T01:13:00.377 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:13:00.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:00.378 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2024-04-22T01:13:00.395 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T01:13:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f0 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f1 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f0 2024-04-22T01:13:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-22T01:13:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 41945sec preferred_lft 41945sec 2024-04-22T01:13:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-22T01:13:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T01:13:00.397 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T01:13:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2024-04-22T01:13:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f1 2024-04-22T01:13:00.397 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T01:13:00.397 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:13:00.397 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:13:00.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2024-04-22T01:13:00.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2024-04-22T01:13:00.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2024-04-22T01:13:00.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T01:13:00.398 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:13:00.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:13:00.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:00.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:00.542 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T01:13:00.565 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-04-22T01:13:00.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:00.567 DEBUG:teuthology.orchestra.run.smithi038:> route 2024-04-22T01:13:00.623 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2024-04-22T01:13:00.623 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T01:13:00.623 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-22T01:13:00.624 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-22T01:13:00.624 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T01:13:00.624 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:13:00.624 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:13:00.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-22T01:13:00.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-22T01:13:00.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-22T01:13:00.624 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:13:00.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:13:00.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:00.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:00.752 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T01:13:00.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:00.767 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2024-04-22T01:13:00.822 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:13:00.822 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:13:00.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:13:00.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T01:13:00.822 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:13:00.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:13:00.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:00.940 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T01:13:00.940 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:13:00.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:13:00.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:00 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1023725778' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T01:13:01.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:00 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1023725778' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T01:13:01.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:01.054 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T01:13:01.054 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T01:13:01.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2024-04-22T01:13:01.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T01:13:01.054 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T01:13:01.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:13:01.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:01.131 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:13:01.131 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:01.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:00 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1023725778' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T01:13:01.146 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T01:13:01.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T01:13:01.147 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:01.197 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-22T01:13:01.197 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-22T01:13:01.197 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/2169144e-b167-4a60-86e4-9fba173df7da 2024-04-22T01:13:01.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:01.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/2169144e-b167-4a60-86e4-9fba173df7da /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T01:13:01.279 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T01:13:01.279 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T01:13:01.279 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T01:13:01.413 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T01:13:01.413 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T01:13:01.413 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T01:13:01.413 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:3300,172.21.15.111:3300,172.21.15.203:3300" 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.38:3300,172.21.15.111:3300,172.21.15.203:3300:/volumes/qa/sv_1/2169144e-b167-4a60-86e4-9fba173df7da 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,key=0,fsid=caf31b2e-0043-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: You just mounted a file system that supports labels which does not 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T01:13:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T01:13:01.417 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T01:13:01.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:01.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:01.487 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T01:13:01.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T01:13:01.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:01.585 INFO:teuthology.orchestra.run.smithi038.stdout:client15498 2024-04-22T01:13:01.585 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:13:01.585 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:13:01.603 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T01:13:01.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T01:13:01.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:01.700 INFO:teuthology.orchestra.run.smithi038.stdout:client15498 2024-04-22T01:13:01.701 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T01:13:01.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T01:13:01.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:01.760 INFO:teuthology.orchestra.run.smithi038.stdout:client15498 2024-04-22T01:13:01.761 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/caf31b2e-0043-11ef-bc93-c7b262605968.client15498/status') 2024-04-22T01:13:01.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T01:13:01.822 INFO:teuthology.orchestra.run.smithi038.stdout:instance: client.15498 (3)192.168.0.1:0/3078018884 2024-04-22T01:13:01.822 INFO:teuthology.orchestra.run.smithi038.stdout:blocklisted: false 2024-04-22T01:13:01.823 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2024-04-22T01:13:01.823 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2024-04-22T01:13:01.823 INFO:teuthology.orchestra.run.smithi038.stderr:70 bytes copied, 2.3795e-05 s, 2.9 MB/s 2024-04-22T01:13:01.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T01:13:01.829 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T01:13:01.850 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T01:13:01.850 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:13:01.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-22T01:13:01.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 -- 172.21.15.38:0/1186537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4941029c0 msgr2=0x7fa494102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:01.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 --2- 172.21.15.38:0/1186537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4941029c0 0x7fa494102e40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa488009900 tx=0x7fa48802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:01.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 -- 172.21.15.38:0/1186537962 shutdown_connections 2024-04-22T01:13:01.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 --2- 172.21.15.38:0/1186537962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494103380 0x7fa494109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 --2- 172.21.15.38:0/1186537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4941029c0 0x7fa494102e40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 --2- 172.21.15.38:0/1186537962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941017d0 0x7fa494101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.953+0000 7fa49891e640 1 -- 172.21.15.38:0/1186537962 >> 172.21.15.38:0/1186537962 conn(0x7fa4940fcd60 msgr2=0x7fa4940ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.954+0000 7fa49891e640 1 -- 172.21.15.38:0/1186537962 shutdown_connections 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.954+0000 7fa49891e640 1 -- 172.21.15.38:0/1186537962 wait complete. 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.954+0000 7fa49891e640 1 Processor -- start 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.954+0000 7fa49891e640 1 -- start start 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.954+0000 7fa49891e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4941017d0 0x7fa494197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.954+0000 7fa49891e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941029c0 0x7fa494197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49891e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 0x7fa49419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49891e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa49410d430 con 0x7fa494103380 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49891e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa49410d2b0 con 0x7fa4941017d0 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49891e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa49410d5b0 con 0x7fa4941029c0 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 0x7fa49419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:01.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 0x7fa49419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51044/0 (socket says 172.21.15.38:51044) 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 -- 172.21.15.38:0/2782325959 learned_addr learned my addr 172.21.15.38:0/2782325959 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49259c640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4941017d0 0x7fa494197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa491d9b640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941029c0 0x7fa494197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 -- 172.21.15.38:0/2782325959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941029c0 msgr2=0x7fa494197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941029c0 0x7fa494197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 -- 172.21.15.38:0/2782325959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4941017d0 msgr2=0x7fa494197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4941017d0 0x7fa494197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 -- 172.21.15.38:0/2782325959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49419f540 con 0x7fa494103380 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa492d9d640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 0x7fa49419eed0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00eaa0 tx=0x7fa48c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00cdb0 con 0x7fa494103380 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49419f830 con 0x7fa494103380 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c005020 con 0x7fa494103380 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0116b0 con 0x7fa494103380 2024-04-22T01:13:01.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.955+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49419fd00 con 0x7fa494103380 2024-04-22T01:13:01.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.957+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa494108a20 con 0x7fa494103380 2024-04-22T01:13:01.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.957+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa48c005560 con 0x7fa494103380 2024-04-22T01:13:01.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.958+0000 7fa4837fe640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa47c0a2c30 0x7fa47c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:01.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.958+0000 7fa49259c640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa47c0a2c30 0x7fa47c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:01.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.958+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa48c0cf690 con 0x7fa494103380 2024-04-22T01:13:01.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.959+0000 7fa49259c640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa47c0a2c30 0x7fa47c0a50f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa494102820 tx=0x7fa484005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:01.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:01.963+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa48c016030 con 0x7fa494103380 2024-04-22T01:13:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:01 smithi203 ceph-mon[31825]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:13:02.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:01 smithi111 ceph-mon[31057]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:13:02.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:01 smithi038 ceph-mon[28043]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T01:13:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.187+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa494062340 con 0x7fa494103380 2024-04-22T01:13:02.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.716+0000 7fa4837fe640 1 -- 172.21.15.38:0/2782325959 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7fa48c09c220 con 0x7fa494103380 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa47c0a2c30 msgr2=0x7fa47c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa47c0a2c30 0x7fa47c0a50f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa494102820 tx=0x7fa484005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 msgr2=0x7fa49419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 0x7fa49419eed0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00eaa0 tx=0x7fa48c00ef60 comp rx=0 tx=0).stop 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 shutdown_connections 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa47c0a2c30 0x7fa47c0a50f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494103380 0x7fa49419eed0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941029c0 0x7fa494197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 --2- 172.21.15.38:0/2782325959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4941017d0 0x7fa494197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.719+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 >> 172.21.15.38:0/2782325959 conn(0x7fa4940fcd60 msgr2=0x7fa494105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:02.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.720+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 shutdown_connections 2024-04-22T01:13:02.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.720+0000 7fa49891e640 1 -- 172.21.15.38:0/2782325959 wait complete. 2024-04-22T01:13:02.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-22T01:13:02.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 -- 172.21.15.38:0/422761657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 msgr2=0x7f925c100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:02.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 --2- 172.21.15.38:0/422761657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c100b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f924c009900 tx=0x7f924c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:02.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 -- 172.21.15.38:0/422761657 shutdown_connections 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 --2- 172.21.15.38:0/422761657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f925c102300 0x7f925c108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 --2- 172.21.15.38:0/422761657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f925c101940 0x7f925c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 --2- 172.21.15.38:0/422761657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c100b70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 -- 172.21.15.38:0/422761657 >> 172.21.15.38:0/422761657 conn(0x7f925c0fbcc0 msgr2=0x7f925c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 -- 172.21.15.38:0/422761657 shutdown_connections 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 -- 172.21.15.38:0/422761657 wait complete. 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.839+0000 7f9262ab5640 1 Processor -- start 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 -- start start 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f925c101940 0x7f925c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f925c102300 0x7f925c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f925c10c360 con 0x7f925c100750 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f925c10c1e0 con 0x7f925c102300 2024-04-22T01:13:02.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9262ab5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f925c10c4e0 con 0x7f925c101940 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51056/0 (socket says 172.21.15.38:51056) 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 -- 172.21.15.38:0/2777954366 learned_addr learned my addr 172.21.15.38:0/2777954366 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9261052640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f925c102300 0x7f925c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 -- 172.21.15.38:0/2777954366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f925c101940 msgr2=0x7f925c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f925c101940 0x7f925c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 -- 172.21.15.38:0/2777954366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f925c102300 msgr2=0x7f925c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f925c102300 0x7f925c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 -- 172.21.15.38:0/2777954366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c1a17f0 con 0x7f925c100750 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9260851640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c199c20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f924c000c00 tx=0x7f924c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.840+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c041070 con 0x7f925c100750 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.841+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f924c0044d0 con 0x7f925c100750 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.841+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c1a1a80 con 0x7f925c100750 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.841+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c034440 con 0x7f925c100750 2024-04-22T01:13:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.841+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c1a1fc0 con 0x7f925c100750 2024-04-22T01:13:02.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.842+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c1a22b0 con 0x7f925c100750 2024-04-22T01:13:02.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.843+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f924c044070 con 0x7f925c100750 2024-04-22T01:13:02.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.843+0000 7f9259ffb640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f92480a2cb0 0x7f92480a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:02.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.844+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f924c0f0f30 con 0x7f925c100750 2024-04-22T01:13:02.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.844+0000 7f925bfff640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f92480a2cb0 0x7f92480a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:02.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.844+0000 7f925bfff640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f92480a2cb0 0x7f92480a5170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f92440058c0 tx=0x7f9244005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:02.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:02.848+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f924c0bdac0 con 0x7f925c100750 2024-04-22T01:13:03.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:02 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2782325959' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T01:13:03.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.063+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f925c0fde40 con 0x7f92480a2cb0 2024-04-22T01:13:03.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:02 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2782325959' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T01:13:03.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.088+0000 7f9259ffb640 1 -- 172.21.15.38:0/2777954366 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f925c0fde40 con 0x7f92480a2cb0 2024-04-22T01:13:03.087 INFO:teuthology.orchestra.run.smithi038.stdout:{} 2024-04-22T01:13:03.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f92480a2cb0 msgr2=0x7f92480a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f92480a2cb0 0x7f92480a5170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f92440058c0 tx=0x7f9244005850 comp rx=0 tx=0).stop 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 msgr2=0x7f925c199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c199c20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f924c000c00 tx=0x7f924c005d50 comp rx=0 tx=0).stop 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 shutdown_connections 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f92480a2cb0 0x7f92480a5170 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f925c102300 0x7f925c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f925c101940 0x7f925c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 --2- 172.21.15.38:0/2777954366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f925c100750 0x7f925c199c20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 >> 172.21.15.38:0/2777954366 conn(0x7f925c0fbcc0 msgr2=0x7f925c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 shutdown_connections 2024-04-22T01:13:03.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.090+0000 7f9262ab5640 1 -- 172.21.15.38:0/2777954366 wait complete. 2024-04-22T01:13:03.100 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-22T01:13:03.113 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-22T01:13:03.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:13:03.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:02 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2782325959' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T01:13:03.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 -- 172.21.15.38:0/170582413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec101880 msgr2=0x7fe5ec101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 --2- 172.21.15.38:0/170582413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec101880 0x7fe5ec101d00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009900 tx=0x7fe5e002bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:03.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 -- 172.21.15.38:0/170582413 shutdown_connections 2024-04-22T01:13:03.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 --2- 172.21.15.38:0/170582413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5ec102240 0x7fe5ec108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 --2- 172.21.15.38:0/170582413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec101880 0x7fe5ec101d00 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 --2- 172.21.15.38:0/170582413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec100690 0x7fe5ec100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 -- 172.21.15.38:0/170582413 >> 172.21.15.38:0/170582413 conn(0x7fe5ec0fbc00 msgr2=0x7fe5ec0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 -- 172.21.15.38:0/170582413 shutdown_connections 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 -- 172.21.15.38:0/170582413 wait complete. 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 Processor -- start 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.230+0000 7fe5f438c640 1 -- start start 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 0x7fe5ec06a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec101880 0x7fe5ec19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5ec102240 0x7fe5ec19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec10c2f0 con 0x7fe5ec100690 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec10c170 con 0x7fe5ec101880 2024-04-22T01:13:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec10c470 con 0x7fe5ec102240 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 0x7fe5ec06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 0x7fe5ec06a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51098/0 (socket says 172.21.15.38:51098) 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 -- 172.21.15.38:0/2819220907 learned_addr learned my addr 172.21.15.38:0/2819220907 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f1927640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec101880 0x7fe5ec19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2929640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5ec102240 0x7fe5ec19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 -- 172.21.15.38:0/2819220907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5ec102240 msgr2=0x7fe5ec19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5ec102240 0x7fe5ec19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 -- 172.21.15.38:0/2819220907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec101880 msgr2=0x7fe5ec19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec101880 0x7fe5ec19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 -- 172.21.15.38:0/2819220907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec19f730 con 0x7fe5ec100690 2024-04-22T01:13:03.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f1927640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec101880 0x7fe5ec19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f2128640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 0x7fe5ec06a990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400e9c0 tx=0x7fe5e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e400cde0 con 0x7fe5ec100690 2024-04-22T01:13:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec19f9c0 con 0x7fe5ec100690 2024-04-22T01:13:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5e4010910 con 0x7fe5ec100690 2024-04-22T01:13:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4018a10 con 0x7fe5ec100690 2024-04-22T01:13:03.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.231+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec19fee0 con 0x7fe5ec100690 2024-04-22T01:13:03.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.232+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec1079b0 con 0x7fe5ec100690 2024-04-22T01:13:03.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.233+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe5e401f020 con 0x7fe5ec100690 2024-04-22T01:13:03.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.234+0000 7fe5eb7fe640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe5d40a2b90 0x7fe5d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.234+0000 7fe5f1927640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe5d40a2b90 0x7fe5d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.234+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe5e40cf620 con 0x7fe5ec100690 2024-04-22T01:13:03.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.235+0000 7fe5f1927640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe5d40a2b90 0x7fe5d40a5050 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe5e002c7b0 tx=0x7fe5e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:03.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.238+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe5e409c1b0 con 0x7fe5ec100690 2024-04-22T01:13:03.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:13:03.068+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:13:03.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:13:03.068+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:13:03.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:13:03.068+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:13:03.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:13:03.068+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:13:03.405 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:13:03.068+0000 7fc4214fd700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T01:13:03.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.423+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5ec0623e0 con 0x7fe5ec100690 2024-04-22T01:13:03.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.425+0000 7fe5eb7fe640 1 -- 172.21.15.38:0/2819220907 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7fe5e409bab0 con 0x7fe5ec100690 2024-04-22T01:13:03.424 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:03.425 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe5d40a2b90 msgr2=0x7fe5d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe5d40a2b90 0x7fe5d40a5050 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe5e002c7b0 tx=0x7fe5e0005fd0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 msgr2=0x7fe5ec06a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 0x7fe5ec06a990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400e9c0 tx=0x7fe5e400ee80 comp rx=0 tx=0).stop 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 shutdown_connections 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe5d40a2b90 0x7fe5d40a5050 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5ec102240 0x7fe5ec19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ec101880 0x7fe5ec19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 --2- 172.21.15.38:0/2819220907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec100690 0x7fe5ec06a990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 >> 172.21.15.38:0/2819220907 conn(0x7fe5ec0fbc00 msgr2=0x7fe5ec104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 shutdown_connections 2024-04-22T01:13:03.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.427+0000 7fe5f438c640 1 -- 172.21.15.38:0/2819220907 wait complete. 2024-04-22T01:13:03.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:13:03.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:13:03.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 -- 172.21.15.38:0/1873531773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec0ffa60 msgr2=0x7f17ec0ffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 --2- 172.21.15.38:0/1873531773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec0ffa60 0x7f17ec0ffe80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009930 tx=0x7f17dc02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:03.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 -- 172.21.15.38:0/1873531773 shutdown_connections 2024-04-22T01:13:03.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 --2- 172.21.15.38:0/1873531773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec1033f0 0x7f17ec109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 --2- 172.21.15.38:0/1873531773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec102a30 0x7f17ec102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 --2- 172.21.15.38:0/1873531773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec0ffa60 0x7f17ec0ffe80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 -- 172.21.15.38:0/1873531773 >> 172.21.15.38:0/1873531773 conn(0x7f17ec0fb370 msgr2=0x7f17ec0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 -- 172.21.15.38:0/1873531773 shutdown_connections 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 -- 172.21.15.38:0/1873531773 wait complete. 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 Processor -- start 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.545+0000 7f17f2678640 1 -- start start 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f2678640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec0ffa60 0x7f17ec1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f2678640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 0x7f17ec1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f2678640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1033f0 0x7f17ec100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f2678640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec10d430 con 0x7f17ec102a30 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f2678640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec10d2b0 con 0x7f17ec0ffa60 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f2678640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec10d5b0 con 0x7f17ec1033f0 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 0x7f17ec1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec0ffa60 0x7f17ec1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 0x7f17ec1009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51116/0 (socket says 172.21.15.38:51116) 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17eb7fe640 1 -- 172.21.15.38:0/3154838619 learned_addr learned my addr 172.21.15.38:0/3154838619 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17f0c15640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1033f0 0x7f17ec100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 -- 172.21.15.38:0/3154838619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1033f0 msgr2=0x7f17ec100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1033f0 0x7f17ec100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 -- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 msgr2=0x7f17ec1009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 0x7f17ec1009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 -- 172.21.15.38:0/3154838619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec1017b0 con 0x7f17ec0ffa60 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 0x7f17ec1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.546+0000 7f17ebfff640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec0ffa60 0x7f17ec1023a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f17dc036040 tx=0x7f17dc02c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.547+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc041070 con 0x7f17ec0ffa60 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.547+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec1a1bf0 con 0x7f17ec0ffa60 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.547+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc004450 con 0x7f17ec0ffa60 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.547+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc003e50 con 0x7f17ec0ffa60 2024-04-22T01:13:03.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.547+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec1a2160 con 0x7f17ec0ffa60 2024-04-22T01:13:03.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.548+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ec108c00 con 0x7f17ec0ffa60 2024-04-22T01:13:03.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.549+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f17dc044070 con 0x7f17ec0ffa60 2024-04-22T01:13:03.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.549+0000 7f17e97fa640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17d40a2d30 0x7f17d40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.550+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f17dc0f11c0 con 0x7f17ec0ffa60 2024-04-22T01:13:03.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.550+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17d40a2d30 0x7f17d40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.550+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17d40a2d30 0x7f17d40a51f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f17ec102110 tx=0x7f17e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:03.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.553+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f17dc0bdd50 con 0x7f17ec0ffa60 2024-04-22T01:13:03.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.738+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f17ec02d3c0 con 0x7f17ec0ffa60 2024-04-22T01:13:03.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.739+0000 7f17e97fa640 1 -- 172.21.15.38:0/3154838619 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f17dc0bd650 con 0x7f17ec0ffa60 2024-04-22T01:13:03.738 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:13:03.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17d40a2d30 msgr2=0x7f17d40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17d40a2d30 0x7f17d40a51f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f17ec102110 tx=0x7f17e000b410 comp rx=0 tx=0).stop 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec0ffa60 msgr2=0x7f17ec1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec0ffa60 0x7f17ec1023a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f17dc036040 tx=0x7f17dc02c6d0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 shutdown_connections 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17d40a2d30 0x7f17d40a51f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1033f0 0x7f17ec100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec102a30 0x7f17ec1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 --2- 172.21.15.38:0/3154838619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec0ffa60 0x7f17ec1023a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 >> 172.21.15.38:0/3154838619 conn(0x7f17ec0fb370 msgr2=0x7f17ec105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 shutdown_connections 2024-04-22T01:13:03.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.741+0000 7f17f2678640 1 -- 172.21.15.38:0/3154838619 wait complete. 2024-04-22T01:13:03.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:13:03.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 -- 172.21.15.38:0/1925623296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 msgr2=0x7f8414108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 --2- 172.21.15.38:0/1925623296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 0x7f8414108b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f840c009be0 tx=0x7f840c02bef0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 -- 172.21.15.38:0/1925623296 shutdown_connections 2024-04-22T01:13:03.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 --2- 172.21.15.38:0/1925623296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 0x7f8414108b40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 --2- 172.21.15.38:0/1925623296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84141008e0 0x7f8414100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 --2- 172.21.15.38:0/1925623296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f8414102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 -- 172.21.15.38:0/1925623296 >> 172.21.15.38:0/1925623296 conn(0x7f84140fb370 msgr2=0x7f84140fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 -- 172.21.15.38:0/1925623296 shutdown_connections 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 -- 172.21.15.38:0/1925623296 wait complete. 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 Processor -- start 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.858+0000 7f8419078640 1 -- start start 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84141008e0 0x7f84141979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 0x7f8414197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f841419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f841410c250 con 0x7f8414101d20 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f841410c0d0 con 0x7f84141012a0 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f841410c3d0 con 0x7f84141008e0 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f841419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f841419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51132/0 (socket says 172.21.15.38:51132) 2024-04-22T01:13:03.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 -- 172.21.15.38:0/905953193 learned_addr learned my addr 172.21.15.38:0/905953193 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8412d9d640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84141008e0 0x7f84141979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f840bfff640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 0x7f8414197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 -- 172.21.15.38:0/905953193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84141008e0 msgr2=0x7f84141979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84141008e0 0x7f84141979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 -- 172.21.15.38:0/905953193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 msgr2=0x7f8414197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 0x7f8414197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 -- 172.21.15.38:0/905953193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841419f5e0 con 0x7f8414101d20 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f841359e640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f841419ef70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f840c009d00 tx=0x7f840c0337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c043020 con 0x7f8414101d20 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840c003b80 con 0x7f8414101d20 2024-04-22T01:13:03.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f841419f870 con 0x7f8414101d20 2024-04-22T01:13:03.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c034690 con 0x7f8414101d20 2024-04-22T01:13:03.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.859+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841419fdc0 con 0x7f8414101d20 2024-04-22T01:13:03.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.861+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84141a0020 con 0x7f8414101d20 2024-04-22T01:13:03.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.862+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f840c046020 con 0x7f8414101d20 2024-04-22T01:13:03.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.862+0000 7f8410d99640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84000a2b90 0x7f84000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:03.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.862+0000 7f8412d9d640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84000a2b90 0x7f84000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:03.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.862+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f840c0f1570 con 0x7f8414101d20 2024-04-22T01:13:03.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.863+0000 7f8412d9d640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84000a2b90 0x7f84000a5050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8404009530 tx=0x7f8404009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:03.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:03.867+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f840c04d070 con 0x7f8414101d20 2024-04-22T01:13:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:03 smithi203 ceph-mon[31825]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:13:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:03 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2782325959' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T01:13:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:03 smithi203 ceph-mon[31825]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T01:13:04.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:03 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2819220907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.056+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f84141a03b0 con 0x7f8414101d20 2024-04-22T01:13:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.056+0000 7f8410d99640 1 -- 172.21.15.38:0/905953193 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f840c0be100 con 0x7f8414101d20 2024-04-22T01:13:04.056 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:13:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84000a2b90 msgr2=0x7f84000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84000a2b90 0x7f84000a5050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8404009530 tx=0x7f8404009260 comp rx=0 tx=0).stop 2024-04-22T01:13:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 msgr2=0x7f841419ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f841419ef70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f840c009d00 tx=0x7f840c0337b0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 shutdown_connections 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f84000a2b90 0x7f84000a5050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8414101d20 0x7f841419ef70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84141012a0 0x7f8414197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 --2- 172.21.15.38:0/905953193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84141008e0 0x7f84141979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 >> 172.21.15.38:0/905953193 conn(0x7f84140fb370 msgr2=0x7f84140fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 shutdown_connections 2024-04-22T01:13:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.059+0000 7f8419078640 1 -- 172.21.15.38:0/905953193 wait complete. 2024-04-22T01:13:04.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-mon[28043]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:13:04.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2782325959' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T01:13:04.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-mon[28043]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T01:13:04.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:03 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2819220907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:04.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:13:04.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:03 smithi111 ceph-mon[31057]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:13:04.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:03 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2782325959' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T01:13:04.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:03 smithi111 ceph-mon[31057]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T01:13:04.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:03 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2819220907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:04.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 -- 172.21.15.38:0/3677225031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 msgr2=0x7f40c4109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 --2- 172.21.15.38:0/3677225031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c4109be0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009ca0 tx=0x7f40bc02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 -- 172.21.15.38:0/3677225031 shutdown_connections 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 --2- 172.21.15.38:0/3677225031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c4109be0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 --2- 172.21.15.38:0/3677225031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c4102990 0x7f40c4102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 --2- 172.21.15.38:0/3677225031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c41017a0 0x7f40c4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 -- 172.21.15.38:0/3677225031 >> 172.21.15.38:0/3677225031 conn(0x7f40c40fcd10 msgr2=0x7f40c40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 -- 172.21.15.38:0/3677225031 shutdown_connections 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.175+0000 7f40cc49d640 1 -- 172.21.15.38:0/3677225031 wait complete. 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 Processor -- start 2024-04-22T01:13:04.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 -- start start 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c41017a0 0x7f40c41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c4102990 0x7f40c4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f40c410d420 con 0x7f40c4103350 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f40c410d2a0 con 0x7f40c4102990 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40cc49d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40c410d5a0 con 0x7f40c41017a0 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51162/0 (socket says 172.21.15.38:51162) 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 -- 172.21.15.38:0/4291351407 learned_addr learned my addr 172.21.15.38:0/4291351407 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:04.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40ca239640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c41017a0 0x7f40c41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40c9a38640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c4102990 0x7f40c4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 -- 172.21.15.38:0/4291351407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c41017a0 msgr2=0x7f40c41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c41017a0 0x7f40c41979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 -- 172.21.15.38:0/4291351407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c4102990 msgr2=0x7f40c4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c4102990 0x7f40c4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40caa3a640 1 -- 172.21.15.38:0/4291351407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c419f5f0 con 0x7f40c4103350 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40c9a38640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c4102990 0x7f40c4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.176+0000 7f40ca239640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c41017a0 0x7f40c41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.177+0000 7f40caa3a640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c419ef80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009bf0 tx=0x7f40bc035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:04.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.177+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc039040 con 0x7f40c4103350 2024-04-22T01:13:04.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.177+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40bc044070 con 0x7f40c4103350 2024-04-22T01:13:04.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.177+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c419f880 con 0x7f40c4103350 2024-04-22T01:13:04.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.177+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc0043a0 con 0x7f40c4103350 2024-04-22T01:13:04.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.177+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c419fdc0 con 0x7f40c4103350 2024-04-22T01:13:04.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.178+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c4108a60 con 0x7f40c4103350 2024-04-22T01:13:04.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.179+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f40bc003ae0 con 0x7f40c4103350 2024-04-22T01:13:04.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.180+0000 7f40bb7fe640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f40b00a2c10 0x7f40b00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.180+0000 7f40ca239640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f40b00a2c10 0x7f40b00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.180+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f40bc0f20f0 con 0x7f40c4103350 2024-04-22T01:13:04.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.181+0000 7f40ca239640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f40b00a2c10 0x7f40b00a50d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f40c41027f0 tx=0x7f40c0005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:04.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.184+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f40bc0bec80 con 0x7f40c4103350 2024-04-22T01:13:04.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.368+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f40c402d300 con 0x7f40c4103350 2024-04-22T01:13:04.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.369+0000 7f40bb7fe640 1 -- 172.21.15.38:0/4291351407 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f40bc0be580 con 0x7f40c4103350 2024-04-22T01:13:04.368 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:13:04.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f40b00a2c10 msgr2=0x7f40b00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f40b00a2c10 0x7f40b00a50d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f40c41027f0 tx=0x7f40c0005db0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 msgr2=0x7f40c419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c419ef80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009bf0 tx=0x7f40bc035ae0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 shutdown_connections 2024-04-22T01:13:04.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f40b00a2c10 0x7f40b00a50d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4103350 0x7f40c419ef80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40c4102990 0x7f40c4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 --2- 172.21.15.38:0/4291351407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40c41017a0 0x7f40c41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 >> 172.21.15.38:0/4291351407 conn(0x7f40c40fcd10 msgr2=0x7f40c4105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 shutdown_connections 2024-04-22T01:13:04.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.371+0000 7f40cc49d640 1 -- 172.21.15.38:0/4291351407 wait complete. 2024-04-22T01:13:04.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:13:04.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 -- 172.21.15.38:0/2113353321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804102250 msgr2=0x7fc804108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/2113353321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804102250 0x7fc804108ae0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009ca0 tx=0x7fc7fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 -- 172.21.15.38:0/2113353321 shutdown_connections 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/2113353321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804102250 0x7fc804108ae0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/2113353321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc804101890 0x7fc804101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/2113353321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8041006a0 0x7fc804100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.487+0000 7fc80c2c0640 1 -- 172.21.15.38:0/2113353321 >> 172.21.15.38:0/2113353321 conn(0x7fc8040fbc30 msgr2=0x7fc8040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 -- 172.21.15.38:0/2113353321 shutdown_connections 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 -- 172.21.15.38:0/2113353321 wait complete. 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 Processor -- start 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 -- start start 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8041006a0 0x7fc804197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 0x7fc804197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804102250 0x7fc80419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc80410c320 con 0x7fc804101890 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc80410c1a0 con 0x7fc8041006a0 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80c2c0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc80410c4a0 con 0x7fc804102250 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.488+0000 7fc80985b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 0x7fc804197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 0x7fc804197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51190/0 (socket says 172.21.15.38:51190) 2024-04-22T01:13:04.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 -- 172.21.15.38:0/1057671244 learned_addr learned my addr 172.21.15.38:0/1057671244 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80a05c640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8041006a0 0x7fc804197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80a85d640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804102250 0x7fc80419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 -- 172.21.15.38:0/1057671244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804102250 msgr2=0x7fc80419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804102250 0x7fc80419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 -- 172.21.15.38:0/1057671244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8041006a0 msgr2=0x7fc804197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8041006a0 0x7fc804197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 -- 172.21.15.38:0/1057671244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80419f6d0 con 0x7fc804101890 2024-04-22T01:13:04.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80a05c640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8041006a0 0x7fc804197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80985b640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 0x7fc804197ff0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400d950 tx=0x7fc7f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4017070 con 0x7fc804101890 2024-04-22T01:13:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f4009c30 con 0x7fc804101890 2024-04-22T01:13:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80419f9c0 con 0x7fc804101890 2024-04-22T01:13:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f400f6b0 con 0x7fc804101890 2024-04-22T01:13:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.489+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80419ff60 con 0x7fc804101890 2024-04-22T01:13:04.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.490+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8041a0190 con 0x7fc804101890 2024-04-22T01:13:04.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.491+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc7f40109d0 con 0x7fc804101890 2024-04-22T01:13:04.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.492+0000 7fc7fb7fe640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc7f00a2c10 0x7fc7f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.492+0000 7fc80a05c640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc7f00a2c10 0x7fc7f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.492+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fc7f40cf0f0 con 0x7fc804101890 2024-04-22T01:13:04.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.492+0000 7fc80a05c640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc7f00a2c10 0x7fc7f00a50d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc8041016f0 tx=0x7fc800005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:04.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.496+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc7f4009de0 con 0x7fc804101890 2024-04-22T01:13:04.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.682+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc8041a0400 con 0x7fc804101890 2024-04-22T01:13:04.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.682+0000 7fc7fb7fe640 1 -- 172.21.15.38:0/1057671244 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fc7f409bc40 con 0x7fc804101890 2024-04-22T01:13:04.682 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc7f00a2c10 msgr2=0x7fc7f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc7f00a2c10 0x7fc7f00a50d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc8041016f0 tx=0x7fc800005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 msgr2=0x7fc804197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 0x7fc804197ff0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400d950 tx=0x7fc7f400de10 comp rx=0 tx=0).stop 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 shutdown_connections 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fc7f00a2c10 0x7fc7f00a50d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804102250 0x7fc80419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc804101890 0x7fc804197ff0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 --2- 172.21.15.38:0/1057671244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8041006a0 0x7fc804197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.684+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 >> 172.21.15.38:0/1057671244 conn(0x7fc8040fbc30 msgr2=0x7fc8040fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:04.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.685+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 shutdown_connections 2024-04-22T01:13:04.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.685+0000 7fc80c2c0640 1 -- 172.21.15.38:0/1057671244 wait complete. 2024-04-22T01:13:04.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:13:04.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 -- 172.21.15.38:0/70134804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa81006a0 msgr2=0x7fbaa8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/70134804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa81006a0 0x7fbaa8100ac0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fba98009900 tx=0x7fba9802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:04.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 -- 172.21.15.38:0/70134804 shutdown_connections 2024-04-22T01:13:04.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/70134804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa8102250 0x7fbaa8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/70134804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8101890 0x7fbaa8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/70134804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa81006a0 0x7fbaa8100ac0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 -- 172.21.15.38:0/70134804 >> 172.21.15.38:0/70134804 conn(0x7fbaa80fbc30 msgr2=0x7fbaa80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:04.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 -- 172.21.15.38:0/70134804 shutdown_connections 2024-04-22T01:13:04.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.803+0000 7fbaad1c8640 1 -- 172.21.15.38:0/70134804 wait complete. 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 Processor -- start 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 -- start start 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa81006a0 0x7fbaa8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 0x7fbaa81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8102250 0x7fbaa819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa810c370 con 0x7fbaa8101890 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa810c1f0 con 0x7fbaa8102250 2024-04-22T01:13:04.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaad1c8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa810c4f0 con 0x7fbaa81006a0 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 0x7fbaa81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 0x7fbaa81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51208/0 (socket says 172.21.15.38:51208) 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 -- 172.21.15.38:0/3104942641 learned_addr learned my addr 172.21.15.38:0/3104942641 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa6d9d640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa81006a0 0x7fbaa8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa759e640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8102250 0x7fbaa819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 -- 172.21.15.38:0/3104942641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa81006a0 msgr2=0x7fbaa8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa81006a0 0x7fbaa8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 -- 172.21.15.38:0/3104942641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8102250 msgr2=0x7fbaa819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8102250 0x7fbaa819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa659c640 1 -- 172.21.15.38:0/3104942641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa819f780 con 0x7fbaa8101890 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.804+0000 7fbaa759e640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8102250 0x7fbaa819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.805+0000 7fbaa659c640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 0x7fbaa81980a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00b4a0 tx=0x7fba9c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.805+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c014070 con 0x7fbaa8101890 2024-04-22T01:13:04.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.805+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa819fa70 con 0x7fbaa8101890 2024-04-22T01:13:04.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.805+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba9c010af0 con 0x7fbaa8101890 2024-04-22T01:13:04.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.805+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c017d10 con 0x7fbaa8101890 2024-04-22T01:13:04.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.805+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa81a0010 con 0x7fbaa8101890 2024-04-22T01:13:04.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.806+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa81a0240 con 0x7fbaa8101890 2024-04-22T01:13:04.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.807+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fba9c017070 con 0x7fbaa8101890 2024-04-22T01:13:04.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.807+0000 7fba8ffff640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fba900a2c20 0x7fba900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:04.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.808+0000 7fbaa6d9d640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fba900a2c20 0x7fba900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:04.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.808+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fba9c0ce840 con 0x7fbaa8101890 2024-04-22T01:13:04.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.808+0000 7fbaa6d9d640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fba900a2c20 0x7fba900a50e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fba98000c00 tx=0x7fba98004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:04.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.812+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fba9c09b340 con 0x7fbaa8101890 2024-04-22T01:13:04.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.997+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbaa81a0410 con 0x7fbaa8101890 2024-04-22T01:13:04.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:04.998+0000 7fba8ffff640 1 -- 172.21.15.38:0/3104942641 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7fba9c09ac40 con 0x7fbaa8101890 2024-04-22T01:13:04.998 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:04.998 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fba900a2c20 msgr2=0x7fba900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fba900a2c20 0x7fba900a50e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fba98000c00 tx=0x7fba98004f10 comp rx=0 tx=0).stop 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 msgr2=0x7fbaa81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 0x7fbaa81980a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00b4a0 tx=0x7fba9c00b960 comp rx=0 tx=0).stop 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 shutdown_connections 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fba900a2c20 0x7fba900a50e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbaa8102250 0x7fbaa819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbaa8101890 0x7fbaa81980a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 --2- 172.21.15.38:0/3104942641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa81006a0 0x7fbaa8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.000+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 >> 172.21.15.38:0/3104942641 conn(0x7fbaa80fbc30 msgr2=0x7fbaa8069dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.001+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 shutdown_connections 2024-04-22T01:13:05.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.001+0000 7fbaad1c8640 1 -- 172.21.15.38:0/3104942641 wait complete. 2024-04-22T01:13:05.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:04 smithi038 ceph-mon[28043]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:13:05.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:04 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3154838619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:04 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/905953193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:04 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4291351407' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:04 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1057671244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.011 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:13:05.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:13:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:04 smithi203 ceph-mon[31825]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:13:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:04 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3154838619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:04 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/905953193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:04 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4291351407' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:04 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1057671244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 -- 172.21.15.38:0/689810075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 msgr2=0x7fa3d4102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/689810075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4102e10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009900 tx=0x7fa3c802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 -- 172.21.15.38:0/689810075 shutdown_connections 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/689810075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d4103350 0x7fa3d4109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/689810075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4102e10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/689810075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d41017a0 0x7fa3d4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 -- 172.21.15.38:0/689810075 >> 172.21.15.38:0/689810075 conn(0x7fa3d40fcd10 msgr2=0x7fa3d40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 -- 172.21.15.38:0/689810075 shutdown_connections 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.117+0000 7fa3da1e5640 1 -- 172.21.15.38:0/689810075 wait complete. 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 Processor -- start 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 -- start start 2024-04-22T01:13:05.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d41017a0 0x7fa3d40754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d4103350 0x7fa3d4074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d410d420 con 0x7fa3d4102990 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d410d2a0 con 0x7fa3d41017a0 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3da1e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d410d5a0 con 0x7fa3d4103350 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3d2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3d2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51234/0 (socket says 172.21.15.38:51234) 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3d2ffd640 1 -- 172.21.15.38:0/3044491467 learned_addr learned my addr 172.21.15.38:0/3044491467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.118+0000 7fa3d3fff640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d4103350 0x7fa3d4074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d37fe640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d41017a0 0x7fa3d40754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d2ffd640 1 -- 172.21.15.38:0/3044491467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d4103350 msgr2=0x7fa3d4074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d2ffd640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d4103350 0x7fa3d4074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d2ffd640 1 -- 172.21.15.38:0/3044491467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d41017a0 msgr2=0x7fa3d40754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d2ffd640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d41017a0 0x7fa3d40754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d2ffd640 1 -- 172.21.15.38:0/3044491467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d4074900 con 0x7fa3d4102990 2024-04-22T01:13:05.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d2ffd640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4073b30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009230 tx=0x7fa3c8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:05.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8041070 con 0x7fa3d4102990 2024-04-22T01:13:05.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d41a1bb0 con 0x7fa3d4102990 2024-04-22T01:13:05.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c80044d0 con 0x7fa3d4102990 2024-04-22T01:13:05.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8034440 con 0x7fa3d4102990 2024-04-22T01:13:05.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.119+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d41a2100 con 0x7fa3d4102990 2024-04-22T01:13:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.120+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d4108ae0 con 0x7fa3d4102990 2024-04-22T01:13:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.121+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fa3c8044070 con 0x7fa3d4102990 2024-04-22T01:13:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.122+0000 7fa3d0ff9640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3bc0a2c20 0x7fa3bc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.122+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa3c80f10e0 con 0x7fa3d4102990 2024-04-22T01:13:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.122+0000 7fa3d37fe640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3bc0a2c20 0x7fa3bc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.122+0000 7fa3d37fe640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3bc0a2c20 0x7fa3bc0a50e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa3d41027f0 tx=0x7fa3c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:05.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.126+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa3c80bdc20 con 0x7fa3d4102990 2024-04-22T01:13:05.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:04 smithi111 ceph-mon[31057]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T01:13:05.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:04 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3154838619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:04 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/905953193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:04 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4291351407' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:04 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1057671244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.311+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa3d402d3c0 con 0x7fa3d4102990 2024-04-22T01:13:05.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.312+0000 7fa3d0ff9640 1 -- 172.21.15.38:0/3044491467 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fa3c80bd520 con 0x7fa3d4102990 2024-04-22T01:13:05.311 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:13:05.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3bc0a2c20 msgr2=0x7fa3bc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3bc0a2c20 0x7fa3bc0a50e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa3d41027f0 tx=0x7fa3c4005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:05.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 msgr2=0x7fa3d4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4073b30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009230 tx=0x7fa3c8005d50 comp rx=0 tx=0).stop 2024-04-22T01:13:05.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 shutdown_connections 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa3bc0a2c20 0x7fa3bc0a50e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d4103350 0x7fa3d4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d4102990 0x7fa3d4073b30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 --2- 172.21.15.38:0/3044491467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d41017a0 0x7fa3d40754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 >> 172.21.15.38:0/3044491467 conn(0x7fa3d40fcd10 msgr2=0x7fa3d4105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 shutdown_connections 2024-04-22T01:13:05.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.314+0000 7fa3da1e5640 1 -- 172.21.15.38:0/3044491467 wait complete. 2024-04-22T01:13:05.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 -- 172.21.15.38:0/984384925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff214102250 msgr2=0x7ff214108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 --2- 172.21.15.38:0/984384925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff214102250 0x7ff214108ae0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009900 tx=0x7ff20c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 -- 172.21.15.38:0/984384925 shutdown_connections 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 --2- 172.21.15.38:0/984384925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff214102250 0x7ff214108ae0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 --2- 172.21.15.38:0/984384925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 0x7ff214101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 --2- 172.21.15.38:0/984384925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2141006a0 0x7ff214100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 -- 172.21.15.38:0/984384925 >> 172.21.15.38:0/984384925 conn(0x7ff2140fbc30 msgr2=0x7ff2140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 -- 172.21.15.38:0/984384925 shutdown_connections 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.430+0000 7ff218bec640 1 -- 172.21.15.38:0/984384925 wait complete. 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 Processor -- start 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 -- start start 2024-04-22T01:13:05.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 0x7ff2141968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 0x7ff214196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff214102250 0x7ff21419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff21410c300 con 0x7ff2141006a0 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff21410c180 con 0x7ff214101890 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff218bec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff21410c480 con 0x7ff214102250 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff21259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 0x7ff2141968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff21259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 0x7ff2141968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51264/0 (socket says 172.21.15.38:51264) 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff21259c640 1 -- 172.21.15.38:0/1375532023 learned_addr learned my addr 172.21.15.38:0/1375532023 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff212d9d640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff214102250 0x7ff21419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff211d9b640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 0x7ff214196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.431+0000 7ff21259c640 1 -- 172.21.15.38:0/1375532023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff214102250 msgr2=0x7ff21419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff21259c640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff214102250 0x7ff21419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff21259c640 1 -- 172.21.15.38:0/1375532023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 msgr2=0x7ff214196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff21259c640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 0x7ff214196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff21259c640 1 -- 172.21.15.38:0/1375532023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21419e4d0 con 0x7ff2141006a0 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff212d9d640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff214102250 0x7ff21419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff211d9b640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 0x7ff214196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff21259c640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 0x7ff2141968b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff20400cfd0 tx=0x7ff20400e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20401a070 con 0x7ff2141006a0 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff204003e50 con 0x7ff2141006a0 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff214069da0 con 0x7ff2141006a0 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204004540 con 0x7ff2141006a0 2024-04-22T01:13:05.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.432+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21406a2f0 con 0x7ff2141006a0 2024-04-22T01:13:05.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.433+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2140689b0 con 0x7ff2141006a0 2024-04-22T01:13:05.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.434+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7ff20401d070 con 0x7ff2141006a0 2024-04-22T01:13:05.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.435+0000 7ff2037fe640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff1fc0a2c20 0x7ff1fc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.435+0000 7ff211d9b640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff1fc0a2c20 0x7ff1fc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.435+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff2040d0bf0 con 0x7ff2141006a0 2024-04-22T01:13:05.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.435+0000 7ff211d9b640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff1fc0a2c20 0x7ff1fc0a50e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff214197dc0 tx=0x7ff208005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:05.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.439+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff20409d620 con 0x7ff2141006a0 2024-04-22T01:13:05.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.622+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff214107680 con 0x7ff2141006a0 2024-04-22T01:13:05.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.623+0000 7ff2037fe640 1 -- 172.21.15.38:0/1375532023 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7ff20409cf20 con 0x7ff2141006a0 2024-04-22T01:13:05.622 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff1fc0a2c20 msgr2=0x7ff1fc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff1fc0a2c20 0x7ff1fc0a50e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff214197dc0 tx=0x7ff208005e50 comp rx=0 tx=0).stop 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 msgr2=0x7ff2141968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 0x7ff2141968b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff20400cfd0 tx=0x7ff20400e420 comp rx=0 tx=0).stop 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 shutdown_connections 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff1fc0a2c20 0x7ff1fc0a50e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff214102250 0x7ff21419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff214101890 0x7ff214196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 --2- 172.21.15.38:0/1375532023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff2141006a0 0x7ff2141968b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 >> 172.21.15.38:0/1375532023 conn(0x7ff2140fbc30 msgr2=0x7ff214104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 shutdown_connections 2024-04-22T01:13:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.625+0000 7ff218bec640 1 -- 172.21.15.38:0/1375532023 wait complete. 2024-04-22T01:13:05.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:13:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.742+0000 7fa2057c9640 1 -- 172.21.15.38:0/2510789297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa200102990 msgr2=0x7fa200102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.742+0000 7fa2057c9640 1 --2- 172.21.15.38:0/2510789297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa200102990 0x7fa200102e10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009900 tx=0x7fa1e802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 -- 172.21.15.38:0/2510789297 shutdown_connections 2024-04-22T01:13:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 --2- 172.21.15.38:0/2510789297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa200103350 0x7fa200109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 --2- 172.21.15.38:0/2510789297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa200102990 0x7fa200102e10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 --2- 172.21.15.38:0/2510789297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2001017a0 0x7fa200101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 -- 172.21.15.38:0/2510789297 >> 172.21.15.38:0/2510789297 conn(0x7fa2000fcd10 msgr2=0x7fa2000ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 -- 172.21.15.38:0/2510789297 shutdown_connections 2024-04-22T01:13:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 -- 172.21.15.38:0/2510789297 wait complete. 2024-04-22T01:13:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 Processor -- start 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 -- start start 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 0x7fa200197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa200102990 0x7fa200197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.743+0000 7fa2057c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa200103350 0x7fa20019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa2057c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010d420 con 0x7fa2001017a0 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa2057c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010d2a0 con 0x7fa200103350 2024-04-22T01:13:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa2057c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010d5a0 con 0x7fa200102990 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 0x7fa200197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 0x7fa200197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51280/0 (socket says 172.21.15.38:51280) 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 -- 172.21.15.38:0/3210968870 learned_addr learned my addr 172.21.15.38:0/3210968870 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1fe7fc640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa200102990 0x7fa200197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1ff7fe640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa200103350 0x7fa20019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 -- 172.21.15.38:0/3210968870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa200102990 msgr2=0x7fa200197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa200102990 0x7fa200197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 -- 172.21.15.38:0/3210968870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa200103350 msgr2=0x7fa20019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa200103350 0x7fa20019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 -- 172.21.15.38:0/3210968870 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20019f660 con 0x7fa2001017a0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1feffd640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 0x7fa200197a40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000c970 tx=0x7fa1f000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0007c10 con 0x7fa2001017a0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1f0007db0 con 0x7fa2001017a0 2024-04-22T01:13:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20019f950 con 0x7fa2001017a0 2024-04-22T01:13:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000f940 con 0x7fa2001017a0 2024-04-22T01:13:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.744+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20019fe70 con 0x7fa2001017a0 2024-04-22T01:13:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.745+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2000689b0 con 0x7fa2001017a0 2024-04-22T01:13:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.747+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fa1f000fae0 con 0x7fa2001017a0 2024-04-22T01:13:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.747+0000 7fa1f7fff640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa1ec0a2c20 0x7fa1ec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:05.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.747+0000 7fa1fe7fc640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa1ec0a2c20 0x7fa1ec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:05.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.747+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa1f00cfc50 con 0x7fa2001017a0 2024-04-22T01:13:05.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.748+0000 7fa1fe7fc640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa1ec0a2c20 0x7fa1ec0a50e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005ab0 tx=0x7fa1e8004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:05.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.751+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa1f009c7a0 con 0x7fa2001017a0 2024-04-22T01:13:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:05 smithi038 ceph-mon[28043]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-22T01:13:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:05 smithi038 ceph-mon[28043]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T01:13:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3104942641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3044491467' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:05 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1375532023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:05.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.937+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa20002d1a0 con 0x7fa2001017a0 2024-04-22T01:13:05.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.938+0000 7fa1f7fff640 1 -- 172.21.15.38:0/3210968870 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fa1f009c0a0 con 0x7fa2001017a0 2024-04-22T01:13:05.937 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa1ec0a2c20 msgr2=0x7fa1ec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa1ec0a2c20 0x7fa1ec0a50e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005ab0 tx=0x7fa1e8004f10 comp rx=0 tx=0).stop 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 msgr2=0x7fa200197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 0x7fa200197a40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000c970 tx=0x7fa1f000ce30 comp rx=0 tx=0).stop 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 shutdown_connections 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa1ec0a2c20 0x7fa1ec0a50e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa200103350 0x7fa20019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa200102990 0x7fa200197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 --2- 172.21.15.38:0/3210968870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2001017a0 0x7fa200197a40 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:05.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 >> 172.21.15.38:0/3210968870 conn(0x7fa2000fcd10 msgr2=0x7fa200105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:05.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 shutdown_connections 2024-04-22T01:13:05.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:05.940+0000 7fa2057c9640 1 -- 172.21.15.38:0/3210968870 wait complete. 2024-04-22T01:13:05.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:13:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:05 smithi203 ceph-mon[31825]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-22T01:13:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:05 smithi203 ceph-mon[31825]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T01:13:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:05 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3104942641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:05 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3044491467' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:06.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:05 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1375532023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- 172.21.15.38:0/882588314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 msgr2=0x7f60ec1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/882588314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec1098b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009ca0 tx=0x7f60e402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- 172.21.15.38:0/882588314 shutdown_connections 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/882588314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec1098b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/882588314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 0x7f60ec106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/882588314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ec102200 0x7f60ec1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- 172.21.15.38:0/882588314 >> 172.21.15.38:0/882588314 conn(0x7f60ec0fbbf0 msgr2=0x7f60ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- 172.21.15.38:0/882588314 shutdown_connections 2024-04-22T01:13:06.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- 172.21.15.38:0/882588314 wait complete. 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 Processor -- start 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- start start 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ec102200 0x7f60ec197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 0x7f60ec197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec10d420 con 0x7f60ec107460 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec10d2a0 con 0x7f60ec104b30 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.056+0000 7f60f2fb8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec10d5a0 con 0x7f60ec102200 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51300/0 (socket says 172.21.15.38:51300) 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 -- 172.21.15.38:0/1738362109 learned_addr learned my addr 172.21.15.38:0/1738362109 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f0d54640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ec102200 0x7f60ec197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60ebfff640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 0x7f60ec197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 -- 172.21.15.38:0/1738362109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ec102200 msgr2=0x7f60ec197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ec102200 0x7f60ec197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 -- 172.21.15.38:0/1738362109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 msgr2=0x7f60ec197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 0x7f60ec197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 -- 172.21.15.38:0/1738362109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec19f6b0 con 0x7f60ec107460 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60ebfff640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 0x7f60ec197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f1555640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec19f040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f60e402c700 tx=0x7f60e4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e4039040 con 0x7f60ec107460 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec19f940 con 0x7f60ec107460 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e4044070 con 0x7f60ec107460 2024-04-22T01:13:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e40042a0 con 0x7f60ec107460 2024-04-22T01:13:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.057+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec19fe30 con 0x7f60ec107460 2024-04-22T01:13:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.058+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec0fd6e0 con 0x7f60ec107460 2024-04-22T01:13:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.060+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f60e4003970 con 0x7f60ec107460 2024-04-22T01:13:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.060+0000 7f60e9ffb640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60dc0a2c20 0x7f60dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.060+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f60e40f2070 con 0x7f60ec107460 2024-04-22T01:13:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.060+0000 7f60f0d54640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60dc0a2c20 0x7f60dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.061+0000 7f60f0d54640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60dc0a2c20 0x7f60dc0a50e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f60e0005fd0 tx=0x7f60e0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:06.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.064+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f60e40bec40 con 0x7f60ec107460 2024-04-22T01:13:06.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:05 smithi111 ceph-mon[31057]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-22T01:13:06.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:05 smithi111 ceph-mon[31057]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T01:13:06.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:05 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3104942641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:06.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:05 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3044491467' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:06.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:05 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1375532023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.248+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f60ec02d170 con 0x7f60ec107460 2024-04-22T01:13:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.248+0000 7f60e9ffb640 1 -- 172.21.15.38:0/1738362109 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f60e40be540 con 0x7f60ec107460 2024-04-22T01:13:06.248 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.250+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60dc0a2c20 msgr2=0x7f60dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.250+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60dc0a2c20 0x7f60dc0a50e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f60e0005fd0 tx=0x7f60e0005850 comp rx=0 tx=0).stop 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.250+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 msgr2=0x7f60ec19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.250+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec19f040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f60e402c700 tx=0x7f60e4032f20 comp rx=0 tx=0).stop 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 shutdown_connections 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60dc0a2c20 0x7f60dc0a50e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60ec107460 0x7f60ec19f040 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60ec104b30 0x7f60ec197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 --2- 172.21.15.38:0/1738362109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60ec102200 0x7f60ec197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 >> 172.21.15.38:0/1738362109 conn(0x7f60ec0fbbf0 msgr2=0x7f60ec102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:06.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 shutdown_connections 2024-04-22T01:13:06.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.251+0000 7f60f2fb8640 1 -- 172.21.15.38:0/1738362109 wait complete. 2024-04-22T01:13:06.261 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-22T01:13:06.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 -- 172.21.15.38:0/3989836414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938102280 msgr2=0x7f9938104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 --2- 172.21.15.38:0/3989836414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938102280 0x7f9938104670 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9934009900 tx=0x7f993402bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 -- 172.21.15.38:0/3989836414 shutdown_connections 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 --2- 172.21.15.38:0/3989836414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 0x7f99381098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 --2- 172.21.15.38:0/3989836414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938104bb0 0x7f9938106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 --2- 172.21.15.38:0/3989836414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938102280 0x7f9938104670 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 -- 172.21.15.38:0/3989836414 >> 172.21.15.38:0/3989836414 conn(0x7f99380fbc30 msgr2=0x7f99380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.364+0000 7f993f688640 1 -- 172.21.15.38:0/3989836414 shutdown_connections 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 -- 172.21.15.38:0/3989836414 wait complete. 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 Processor -- start 2024-04-22T01:13:06.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 -- start start 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938102280 0x7f9938197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 0x7f99381980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 0x7f993819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f993810d490 con 0x7f9938104bb0 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f993810d310 con 0x7f99381074e0 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993f688640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f993810d610 con 0x7f9938102280 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993cc23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 0x7f99381980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993cc23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 0x7f99381980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51310/0 (socket says 172.21.15.38:51310) 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.365+0000 7f993cc23640 1 -- 172.21.15.38:0/2278702210 learned_addr learned my addr 172.21.15.38:0/2278702210 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993d424640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938102280 0x7f9938197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993cc23640 1 -- 172.21.15.38:0/2278702210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938102280 msgr2=0x7f9938197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993dc25640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 0x7f993819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993cc23640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938102280 0x7f9938197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993cc23640 1 -- 172.21.15.38:0/2278702210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 msgr2=0x7f993819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993cc23640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 0x7f993819f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993cc23640 1 -- 172.21.15.38:0/2278702210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993819f790 con 0x7f9938104bb0 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993d424640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938102280 0x7f9938197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993dc25640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 0x7f993819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993cc23640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 0x7f99381980b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f992800b450 tx=0x7f992800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928015020 con 0x7f9938104bb0 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993819fa80 con 0x7f9938104bb0 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9928011d70 con 0x7f9938104bb0 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992800f340 con 0x7f9938104bb0 2024-04-22T01:13:06.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.366+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99381a0020 con 0x7f9938104bb0 2024-04-22T01:13:06.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.367+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99380fd720 con 0x7f9938104bb0 2024-04-22T01:13:06.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.368+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f9928018020 con 0x7f9938104bb0 2024-04-22T01:13:06.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.369+0000 7f992e7fc640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99240a2c20 0x7f99240a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.369+0000 7f993d424640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99240a2c20 0x7f99240a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.369+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f99280cf500 con 0x7f9938104bb0 2024-04-22T01:13:06.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.370+0000 7f993d424640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99240a2c20 0x7f99240a50e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9934036040 tx=0x7f9934000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:06.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.373+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f992801f020 con 0x7f9938104bb0 2024-04-22T01:13:06.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.596+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f993802d170 con 0x7f9938104bb0 2024-04-22T01:13:06.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.597+0000 7f992e7fc640 1 -- 172.21.15.38:0/2278702210 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f992800d040 con 0x7f9938104bb0 2024-04-22T01:13:06.596 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:06.596 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:13:06.596 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:13:06.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99240a2c20 msgr2=0x7f99240a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99240a2c20 0x7f99240a50e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9934036040 tx=0x7f9934000bc0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 msgr2=0x7f99381980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 0x7f99381980b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f992800b450 tx=0x7f992800b910 comp rx=0 tx=0).stop 2024-04-22T01:13:06.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 shutdown_connections 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f99240a2c20 0x7f99240a50e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99381074e0 0x7f993819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9938104bb0 0x7f99381980b0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 --2- 172.21.15.38:0/2278702210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9938102280 0x7f9938197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 >> 172.21.15.38:0/2278702210 conn(0x7f99380fbc30 msgr2=0x7f9938102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 shutdown_connections 2024-04-22T01:13:06.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.599+0000 7f993f688640 1 -- 172.21.15.38:0/2278702210 wait complete. 2024-04-22T01:13:06.609 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-22T01:13:06.609 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-22T01:13:06.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 -- 172.21.15.38:0/1372719967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 msgr2=0x7f2a90108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 --2- 172.21.15.38:0/1372719967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90108ac0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009ca0 tx=0x7f2a8802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 -- 172.21.15.38:0/1372719967 shutdown_connections 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 --2- 172.21.15.38:0/1372719967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90108ac0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 --2- 172.21.15.38:0/1372719967 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a90101870 0x7f2a90101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 --2- 172.21.15.38:0/1372719967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 0x7f2a90100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 -- 172.21.15.38:0/1372719967 >> 172.21.15.38:0/1372719967 conn(0x7f2a900fbbf0 msgr2=0x7f2a900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.721+0000 7f2a984a7640 1 -- 172.21.15.38:0/1372719967 shutdown_connections 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 -- 172.21.15.38:0/1372719967 wait complete. 2024-04-22T01:13:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 Processor -- start 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 -- start start 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 0x7f2a90104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a90101870 0x7f2a90102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9010c300 con 0x7f2a90102230 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9010c180 con 0x7f2a90100680 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a984a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9010c480 con 0x7f2a90101870 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a96a44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a96a44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37292/0 (socket says 172.21.15.38:37292) 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.722+0000 7f2a96a44640 1 -- 172.21.15.38:0/366282910 learned_addr learned my addr 172.21.15.38:0/366282910 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96243640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 0x7f2a90104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a95a42640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a90101870 0x7f2a90102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96a44640 1 -- 172.21.15.38:0/366282910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a90101870 msgr2=0x7f2a90102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96a44640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a90101870 0x7f2a90102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96a44640 1 -- 172.21.15.38:0/366282910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 msgr2=0x7f2a90104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96a44640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 0x7f2a90104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96a44640 1 -- 172.21.15.38:0/366282910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a90103a20 con 0x7f2a90102230 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96243640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 0x7f2a90104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a96a44640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90103190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2a88037040 tx=0x7f2a88032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88041020 con 0x7f2a90102230 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a88034cd0 con 0x7f2a90102230 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88039040 con 0x7f2a90102230 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a901a1b50 con 0x7f2a90102230 2024-04-22T01:13:06.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.723+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a901a20a0 con 0x7f2a90102230 2024-04-22T01:13:06.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.724+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a901a2270 con 0x7f2a90102230 2024-04-22T01:13:06.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.726+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f2a88044020 con 0x7f2a90102230 2024-04-22T01:13:06.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.727+0000 7f2a837fe640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2a7c0a2c20 0x7f2a7c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:06.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.727+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f2a880f1300 con 0x7f2a90102230 2024-04-22T01:13:06.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.727+0000 7f2a96243640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2a7c0a2c20 0x7f2a7c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:06.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.727+0000 7f2a96243640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2a7c0a2c20 0x7f2a7c0a50e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2a901016d0 tx=0x7f2a84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:06.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.731+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2a8804b020 con 0x7f2a90102230 2024-04-22T01:13:06.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3210968870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:06.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1738362109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:06.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:06 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2278702210' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:06.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:06 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:13:06.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.917+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a901a2440 con 0x7f2a7c0a2c20 2024-04-22T01:13:06.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.928+0000 7f2a837fe640 1 -- 172.21.15.38:0/366282910 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340810 (secure 0 0 0) 0x7f2a901a2440 con 0x7f2a7c0a2c20 2024-04-22T01:13:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2a7c0a2c20 msgr2=0x7f2a7c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2a7c0a2c20 0x7f2a7c0a50e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2a901016d0 tx=0x7f2a84005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 msgr2=0x7f2a90103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90103190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2a88037040 tx=0x7f2a88032f40 comp rx=0 tx=0).stop 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 shutdown_connections 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2a7c0a2c20 0x7f2a7c0a50e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a90102230 0x7f2a90103190 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a90101870 0x7f2a90102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 --2- 172.21.15.38:0/366282910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a90100680 0x7f2a90104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:06.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.931+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 >> 172.21.15.38:0/366282910 conn(0x7f2a900fbbf0 msgr2=0x7f2a900fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:06.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.932+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 shutdown_connections 2024-04-22T01:13:06.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:06.932+0000 7f2a984a7640 1 -- 172.21.15.38:0/366282910 wait complete. 2024-04-22T01:13:06.943 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-04-22T01:13:05.826500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660377,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1382,"num_read_kb":809,"num_write":440,"num_write_kb":3091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181408,"num_keys_recovered":7,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357636,"kb_used_data":12592,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584188,"statfs":{"total":1151940427776,"available":1151574208512,"internally_reserved":0,"allocated":12894208,"data_stored":8202845,"data_compressed":428511,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1398,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":52,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790906+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:12:17.790906+0000","last_peered":"2024-04-22T01:12:17.790906+0000","last_clean":"2024-04-22T01:12:17.790906+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:12:17.790906+0000","last_undegraded":"2024-04-22T01:12:17.790906+0000","last_fullsized":"2024-04-22T01:12:17.790906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468337+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:12:17.468337+0000","last_peered":"2024-04-22T01:12:17.468337+0000","last_clean":"2024-04-22T01:12:17.468337+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:12:17.468337+0000","last_undegraded":"2024-04-22T01:12:17.468337+0000","last_fullsized":"2024-04-22T01:12:17.468337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467910+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:12:17.467910+0000","last_peered":"2024-04-22T01:12:17.467910+0000","last_clean":"2024-04-22T01:12:17.467910+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:12:17.467910+0000","last_undegraded":"2024-04-22T01:12:17.467910+0000","last_fullsized":"2024-04-22T01:12:17.467910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464881+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:12:17.464881+0000","last_peered":"2024-04-22T01:12:17.464881+0000","last_clean":"2024-04-22T01:12:17.464881+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:12:17.464881+0000","last_undegraded":"2024-04-22T01:12:17.464881+0000","last_fullsized":"2024-04-22T01:12:17.464881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467393+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:12:17.467393+0000","last_peered":"2024-04-22T01:12:17.467393+0000","last_clean":"2024-04-22T01:12:17.467393+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:12:17.467393+0000","last_undegraded":"2024-04-22T01:12:17.467393+0000","last_fullsized":"2024-04-22T01:12:17.467393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466033+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:12:17.466033+0000","last_peered":"2024-04-22T01:12:17.466033+0000","last_clean":"2024-04-22T01:12:17.466033+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:12:17.466033+0000","last_undegraded":"2024-04-22T01:12:17.466033+0000","last_fullsized":"2024-04-22T01:12:17.466033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467329+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:12:17.467329+0000","last_peered":"2024-04-22T01:12:17.467329+0000","last_clean":"2024-04-22T01:12:17.467329+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:12:17.467329+0000","last_undegraded":"2024-04-22T01:12:17.467329+0000","last_fullsized":"2024-04-22T01:12:17.467329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467174+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:12:17.467174+0000","last_peered":"2024-04-22T01:12:17.467174+0000","last_clean":"2024-04-22T01:12:17.467174+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:12:17.467174+0000","last_undegraded":"2024-04-22T01:12:17.467174+0000","last_fullsized":"2024-04-22T01:12:17.467174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471434+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:12:17.471434+0000","last_peered":"2024-04-22T01:12:17.471434+0000","last_clean":"2024-04-22T01:12:17.471434+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:12:17.471434+0000","last_undegraded":"2024-04-22T01:12:17.471434+0000","last_fullsized":"2024-04-22T01:12:17.471434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466554+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:12:17.466554+0000","last_peered":"2024-04-22T01:12:17.466554+0000","last_clean":"2024-04-22T01:12:17.466554+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:12:17.466554+0000","last_undegraded":"2024-04-22T01:12:17.466554+0000","last_fullsized":"2024-04-22T01:12:17.466554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468253+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:12:17.468253+0000","last_peered":"2024-04-22T01:12:17.468253+0000","last_clean":"2024-04-22T01:12:17.468253+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:12:17.468253+0000","last_undegraded":"2024-04-22T01:12:17.468253+0000","last_fullsized":"2024-04-22T01:12:17.468253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464762+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:12:17.464762+0000","last_peered":"2024-04-22T01:12:17.464762+0000","last_clean":"2024-04-22T01:12:17.464762+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:12:17.464762+0000","last_undegraded":"2024-04-22T01:12:17.464762+0000","last_fullsized":"2024-04-22T01:12:17.464762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462419+0000","last_change":"2024-04-22T01:11:17.312524+0000","last_active":"2024-04-22T01:12:17.462419+0000","last_peered":"2024-04-22T01:12:17.462419+0000","last_clean":"2024-04-22T01:12:17.462419+0000","last_became_active":"2024-04-22T01:11:17.312068+0000","last_became_peered":"2024-04-22T01:11:17.312068+0000","last_unstale":"2024-04-22T01:12:17.462419+0000","last_undegraded":"2024-04-22T01:12:17.462419+0000","last_fullsized":"2024-04-22T01:12:17.462419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:47:17.243712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467343+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:12:17.467343+0000","last_peered":"2024-04-22T01:12:17.467343+0000","last_clean":"2024-04-22T01:12:17.467343+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:12:17.467343+0000","last_undegraded":"2024-04-22T01:12:17.467343+0000","last_fullsized":"2024-04-22T01:12:17.467343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467849+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:12:17.467849+0000","last_peered":"2024-04-22T01:12:17.467849+0000","last_clean":"2024-04-22T01:12:17.467849+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:12:17.467849+0000","last_undegraded":"2024-04-22T01:12:17.467849+0000","last_fullsized":"2024-04-22T01:12:17.467849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470498+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:12:17.470498+0000","last_peered":"2024-04-22T01:12:17.470498+0000","last_clean":"2024-04-22T01:12:17.470498+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:12:17.470498+0000","last_undegraded":"2024-04-22T01:12:17.470498+0000","last_fullsized":"2024-04-22T01:12:17.470498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790752+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:12:17.790752+0000","last_peered":"2024-04-22T01:12:17.790752+0000","last_clean":"2024-04-22T01:12:17.790752+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:12:17.790752+0000","last_undegraded":"2024-04-22T01:12:17.790752+0000","last_fullsized":"2024-04-22T01:12:17.790752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465060+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:12:17.465060+0000","last_peered":"2024-04-22T01:12:17.465060+0000","last_clean":"2024-04-22T01:12:17.465060+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:12:17.465060+0000","last_undegraded":"2024-04-22T01:12:17.465060+0000","last_fullsized":"2024-04-22T01:12:17.465060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470457+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:12:17.470457+0000","last_peered":"2024-04-22T01:12:17.470457+0000","last_clean":"2024-04-22T01:12:17.470457+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:12:17.470457+0000","last_undegraded":"2024-04-22T01:12:17.470457+0000","last_fullsized":"2024-04-22T01:12:17.470457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470430+0000","last_change":"2024-04-22T01:11:17.586121+0000","last_active":"2024-04-22T01:12:17.470430+0000","last_peered":"2024-04-22T01:12:17.470430+0000","last_clean":"2024-04-22T01:12:17.470430+0000","last_became_active":"2024-04-22T01:11:17.585619+0000","last_became_peered":"2024-04-22T01:11:17.585619+0000","last_unstale":"2024-04-22T01:12:17.470430+0000","last_undegraded":"2024-04-22T01:12:17.470430+0000","last_fullsized":"2024-04-22T01:12:17.470430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:12.987810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:59.529087+0000","last_change":"2024-04-22T01:12:17.484778+0000","last_active":"2024-04-22T01:12:59.529087+0000","last_peered":"2024-04-22T01:12:59.529087+0000","last_clean":"2024-04-22T01:12:59.529087+0000","last_became_active":"2024-04-22T01:12:17.472811+0000","last_became_peered":"2024-04-22T01:12:17.472811+0000","last_unstale":"2024-04-22T01:12:59.529087+0000","last_undegraded":"2024-04-22T01:12:59.529087+0000","last_fullsized":"2024-04-22T01:12:59.529087+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:07.118642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467391+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:12:17.467391+0000","last_peered":"2024-04-22T01:12:17.467391+0000","last_clean":"2024-04-22T01:12:17.467391+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:12:17.467391+0000","last_undegraded":"2024-04-22T01:12:17.467391+0000","last_fullsized":"2024-04-22T01:12:17.467391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464207+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:12:17.464207+0000","last_peered":"2024-04-22T01:12:17.464207+0000","last_clean":"2024-04-22T01:12:17.464207+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:12:17.464207+0000","last_undegraded":"2024-04-22T01:12:17.464207+0000","last_fullsized":"2024-04-22T01:12:17.464207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468644+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:12:17.468644+0000","last_peered":"2024-04-22T01:12:17.468644+0000","last_clean":"2024-04-22T01:12:17.468644+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:12:17.468644+0000","last_undegraded":"2024-04-22T01:12:17.468644+0000","last_fullsized":"2024-04-22T01:12:17.468644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468490+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:12:17.468490+0000","last_peered":"2024-04-22T01:12:17.468490+0000","last_clean":"2024-04-22T01:12:17.468490+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:12:17.468490+0000","last_undegraded":"2024-04-22T01:12:17.468490+0000","last_fullsized":"2024-04-22T01:12:17.468490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464299+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:12:17.464299+0000","last_peered":"2024-04-22T01:12:17.464299+0000","last_clean":"2024-04-22T01:12:17.464299+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:12:17.464299+0000","last_undegraded":"2024-04-22T01:12:17.464299+0000","last_fullsized":"2024-04-22T01:12:17.464299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467663+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:12:17.467663+0000","last_peered":"2024-04-22T01:12:17.467663+0000","last_clean":"2024-04-22T01:12:17.467663+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:12:17.467663+0000","last_undegraded":"2024-04-22T01:12:17.467663+0000","last_fullsized":"2024-04-22T01:12:17.467663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470754+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:12:17.470754+0000","last_peered":"2024-04-22T01:12:17.470754+0000","last_clean":"2024-04-22T01:12:17.470754+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:12:17.470754+0000","last_undegraded":"2024-04-22T01:12:17.470754+0000","last_fullsized":"2024-04-22T01:12:17.470754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464253+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:12:17.464253+0000","last_peered":"2024-04-22T01:12:17.464253+0000","last_clean":"2024-04-22T01:12:17.464253+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:12:17.464253+0000","last_undegraded":"2024-04-22T01:12:17.464253+0000","last_fullsized":"2024-04-22T01:12:17.464253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467272+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:12:17.467272+0000","last_peered":"2024-04-22T01:12:17.467272+0000","last_clean":"2024-04-22T01:12:17.467272+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:12:17.467272+0000","last_undegraded":"2024-04-22T01:12:17.467272+0000","last_fullsized":"2024-04-22T01:12:17.467272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463194+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:12:17.463194+0000","last_peered":"2024-04-22T01:12:17.463194+0000","last_clean":"2024-04-22T01:12:17.463194+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:12:17.463194+0000","last_undegraded":"2024-04-22T01:12:17.463194+0000","last_fullsized":"2024-04-22T01:12:17.463194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470258+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:12:17.470258+0000","last_peered":"2024-04-22T01:12:17.470258+0000","last_clean":"2024-04-22T01:12:17.470258+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:12:17.470258+0000","last_undegraded":"2024-04-22T01:12:17.470258+0000","last_fullsized":"2024-04-22T01:12:17.470258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:03.625138+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:13:03.625138+0000","last_peered":"2024-04-22T01:13:03.625138+0000","last_clean":"2024-04-22T01:13:03.625138+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:13:03.625138+0000","last_undegraded":"2024-04-22T01:13:03.625138+0000","last_fullsized":"2024-04-22T01:13:03.625138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466990+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:12:17.466990+0000","last_peered":"2024-04-22T01:12:17.466990+0000","last_clean":"2024-04-22T01:12:17.466990+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:12:17.466990+0000","last_undegraded":"2024-04-22T01:12:17.466990+0000","last_fullsized":"2024-04-22T01:12:17.466990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470301+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:12:17.470301+0000","last_peered":"2024-04-22T01:12:17.470301+0000","last_clean":"2024-04-22T01:12:17.470301+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:12:17.470301+0000","last_undegraded":"2024-04-22T01:12:17.470301+0000","last_fullsized":"2024-04-22T01:12:17.470301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465883+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:12:17.465883+0000","last_peered":"2024-04-22T01:12:17.465883+0000","last_clean":"2024-04-22T01:12:17.465883+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:12:17.465883+0000","last_undegraded":"2024-04-22T01:12:17.465883+0000","last_fullsized":"2024-04-22T01:12:17.465883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463275+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:12:17.463275+0000","last_peered":"2024-04-22T01:12:17.463275+0000","last_clean":"2024-04-22T01:12:17.463275+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:12:17.463275+0000","last_undegraded":"2024-04-22T01:12:17.463275+0000","last_fullsized":"2024-04-22T01:12:17.463275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790698+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:12:17.790698+0000","last_peered":"2024-04-22T01:12:17.790698+0000","last_clean":"2024-04-22T01:12:17.790698+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:12:17.790698+0000","last_undegraded":"2024-04-22T01:12:17.790698+0000","last_fullsized":"2024-04-22T01:12:17.790698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.461932+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:12:17.461932+0000","last_peered":"2024-04-22T01:12:17.461932+0000","last_clean":"2024-04-22T01:12:17.461932+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:12:17.461932+0000","last_undegraded":"2024-04-22T01:12:17.461932+0000","last_fullsized":"2024-04-22T01:12:17.461932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467441+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:12:17.467441+0000","last_peered":"2024-04-22T01:12:17.467441+0000","last_clean":"2024-04-22T01:12:17.467441+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:12:17.467441+0000","last_undegraded":"2024-04-22T01:12:17.467441+0000","last_fullsized":"2024-04-22T01:12:17.467441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468371+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:12:17.468371+0000","last_peered":"2024-04-22T01:12:17.468371+0000","last_clean":"2024-04-22T01:12:17.468371+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:12:17.468371+0000","last_undegraded":"2024-04-22T01:12:17.468371+0000","last_fullsized":"2024-04-22T01:12:17.468371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462747+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:12:17.462747+0000","last_peered":"2024-04-22T01:12:17.462747+0000","last_clean":"2024-04-22T01:12:17.462747+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:12:17.462747+0000","last_undegraded":"2024-04-22T01:12:17.462747+0000","last_fullsized":"2024-04-22T01:12:17.462747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464591+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:12:17.464591+0000","last_peered":"2024-04-22T01:12:17.464591+0000","last_clean":"2024-04-22T01:12:17.464591+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:12:17.464591+0000","last_undegraded":"2024-04-22T01:12:17.464591+0000","last_fullsized":"2024-04-22T01:12:17.464591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467587+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:12:17.467587+0000","last_peered":"2024-04-22T01:12:17.467587+0000","last_clean":"2024-04-22T01:12:17.467587+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:12:17.467587+0000","last_undegraded":"2024-04-22T01:12:17.467587+0000","last_fullsized":"2024-04-22T01:12:17.467587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":279,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.619054+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:13:00.619054+0000","last_peered":"2024-04-22T01:13:00.619054+0000","last_clean":"2024-04-22T01:13:00.619054+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:13:00.619054+0000","last_undegraded":"2024-04-22T01:13:00.619054+0000","last_fullsized":"2024-04-22T01:13:00.619054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471276+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:12:17.471276+0000","last_peered":"2024-04-22T01:12:17.471276+0000","last_clean":"2024-04-22T01:12:17.471276+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:12:17.471276+0000","last_undegraded":"2024-04-22T01:12:17.471276+0000","last_fullsized":"2024-04-22T01:12:17.471276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464653+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:12:17.464653+0000","last_peered":"2024-04-22T01:12:17.464653+0000","last_clean":"2024-04-22T01:12:17.464653+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:12:17.464653+0000","last_undegraded":"2024-04-22T01:12:17.464653+0000","last_fullsized":"2024-04-22T01:12:17.464653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.477748+0000","last_change":"2024-04-22T01:12:17.477108+0000","last_active":"2024-04-22T01:12:17.477748+0000","last_peered":"2024-04-22T01:12:17.477748+0000","last_clean":"2024-04-22T01:12:17.477748+0000","last_became_active":"2024-04-22T01:12:17.471865+0000","last_became_peered":"2024-04-22T01:12:17.471865+0000","last_unstale":"2024-04-22T01:12:17.477748+0000","last_undegraded":"2024-04-22T01:12:17.477748+0000","last_fullsized":"2024-04-22T01:12:17.477748+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:49:55.734197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468386+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:12:17.468386+0000","last_peered":"2024-04-22T01:12:17.468386+0000","last_clean":"2024-04-22T01:12:17.468386+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:12:17.468386+0000","last_undegraded":"2024-04-22T01:12:17.468386+0000","last_fullsized":"2024-04-22T01:12:17.468386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464504+0000","last_change":"2024-04-22T01:11:17.586221+0000","last_active":"2024-04-22T01:12:17.464504+0000","last_peered":"2024-04-22T01:12:17.464504+0000","last_clean":"2024-04-22T01:12:17.464504+0000","last_became_active":"2024-04-22T01:11:17.585689+0000","last_became_peered":"2024-04-22T01:11:17.585689+0000","last_unstale":"2024-04-22T01:12:17.464504+0000","last_undegraded":"2024-04-22T01:12:17.464504+0000","last_fullsized":"2024-04-22T01:12:17.464504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:40.327272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463285+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:12:17.463285+0000","last_peered":"2024-04-22T01:12:17.463285+0000","last_clean":"2024-04-22T01:12:17.463285+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:12:17.463285+0000","last_undegraded":"2024-04-22T01:12:17.463285+0000","last_fullsized":"2024-04-22T01:12:17.463285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470885+0000","last_change":"2024-04-22T01:11:17.585668+0000","last_active":"2024-04-22T01:12:17.470885+0000","last_peered":"2024-04-22T01:12:17.470885+0000","last_clean":"2024-04-22T01:12:17.470885+0000","last_became_active":"2024-04-22T01:11:17.585217+0000","last_became_peered":"2024-04-22T01:11:17.585217+0000","last_unstale":"2024-04-22T01:12:17.470885+0000","last_undegraded":"2024-04-22T01:12:17.470885+0000","last_fullsized":"2024-04-22T01:12:17.470885+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:24.007210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464709+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:12:17.464709+0000","last_peered":"2024-04-22T01:12:17.464709+0000","last_clean":"2024-04-22T01:12:17.464709+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:12:17.464709+0000","last_undegraded":"2024-04-22T01:12:17.464709+0000","last_fullsized":"2024-04-22T01:12:17.464709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'38","reported_seq":318,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:01.485731+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:13:01.485731+0000","last_peered":"2024-04-22T01:13:01.485731+0000","last_clean":"2024-04-22T01:13:01.485731+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:13:01.485731+0000","last_undegraded":"2024-04-22T01:13:01.485731+0000","last_fullsized":"2024-04-22T01:13:01.485731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61199,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":249,"num_read_kb":184,"num_write":40,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467503+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:12:17.467503+0000","last_peered":"2024-04-22T01:12:17.467503+0000","last_clean":"2024-04-22T01:12:17.467503+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:12:17.467503+0000","last_undegraded":"2024-04-22T01:12:17.467503+0000","last_fullsized":"2024-04-22T01:12:17.467503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463529+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:12:17.463529+0000","last_peered":"2024-04-22T01:12:17.463529+0000","last_clean":"2024-04-22T01:12:17.463529+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:12:17.463529+0000","last_undegraded":"2024-04-22T01:12:17.463529+0000","last_fullsized":"2024-04-22T01:12:17.463529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470394+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:12:17.470394+0000","last_peered":"2024-04-22T01:12:17.470394+0000","last_clean":"2024-04-22T01:12:17.470394+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:12:17.470394+0000","last_undegraded":"2024-04-22T01:12:17.470394+0000","last_fullsized":"2024-04-22T01:12:17.470394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462066+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:12:17.462066+0000","last_peered":"2024-04-22T01:12:17.462066+0000","last_clean":"2024-04-22T01:12:17.462066+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:12:17.462066+0000","last_undegraded":"2024-04-22T01:12:17.462066+0000","last_fullsized":"2024-04-22T01:12:17.462066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790993+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:12:17.790993+0000","last_peered":"2024-04-22T01:12:17.790993+0000","last_clean":"2024-04-22T01:12:17.790993+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:12:17.790993+0000","last_undegraded":"2024-04-22T01:12:17.790993+0000","last_fullsized":"2024-04-22T01:12:17.790993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463155+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:12:17.463155+0000","last_peered":"2024-04-22T01:12:17.463155+0000","last_clean":"2024-04-22T01:12:17.463155+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:12:17.463155+0000","last_undegraded":"2024-04-22T01:12:17.463155+0000","last_fullsized":"2024-04-22T01:12:17.463155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467829+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:12:17.467829+0000","last_peered":"2024-04-22T01:12:17.467829+0000","last_clean":"2024-04-22T01:12:17.467829+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:12:17.467829+0000","last_undegraded":"2024-04-22T01:12:17.467829+0000","last_fullsized":"2024-04-22T01:12:17.467829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464953+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:12:17.464953+0000","last_peered":"2024-04-22T01:12:17.464953+0000","last_clean":"2024-04-22T01:12:17.464953+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:12:17.464953+0000","last_undegraded":"2024-04-22T01:12:17.464953+0000","last_fullsized":"2024-04-22T01:12:17.464953+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.654949+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:13:00.654949+0000","last_peered":"2024-04-22T01:13:00.654949+0000","last_clean":"2024-04-22T01:13:00.654949+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:13:00.654949+0000","last_undegraded":"2024-04-22T01:13:00.654949+0000","last_fullsized":"2024-04-22T01:13:00.654949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467681+0000","last_change":"2024-04-22T01:11:17.314098+0000","last_active":"2024-04-22T01:12:17.467681+0000","last_peered":"2024-04-22T01:12:17.467681+0000","last_clean":"2024-04-22T01:12:17.467681+0000","last_became_active":"2024-04-22T01:11:17.313663+0000","last_became_peered":"2024-04-22T01:11:17.313663+0000","last_unstale":"2024-04-22T01:12:17.467681+0000","last_undegraded":"2024-04-22T01:12:17.467681+0000","last_fullsized":"2024-04-22T01:12:17.467681+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:20:03.826857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462323+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:12:17.462323+0000","last_peered":"2024-04-22T01:12:17.462323+0000","last_clean":"2024-04-22T01:12:17.462323+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:12:17.462323+0000","last_undegraded":"2024-04-22T01:12:17.462323+0000","last_fullsized":"2024-04-22T01:12:17.462323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466996+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:12:17.466996+0000","last_peered":"2024-04-22T01:12:17.466996+0000","last_clean":"2024-04-22T01:12:17.466996+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:12:17.466996+0000","last_undegraded":"2024-04-22T01:12:17.466996+0000","last_fullsized":"2024-04-22T01:12:17.466996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462862+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:12:17.462862+0000","last_peered":"2024-04-22T01:12:17.462862+0000","last_clean":"2024-04-22T01:12:17.462862+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:12:17.462862+0000","last_undegraded":"2024-04-22T01:12:17.462862+0000","last_fullsized":"2024-04-22T01:12:17.462862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463203+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:12:17.463203+0000","last_peered":"2024-04-22T01:12:17.463203+0000","last_clean":"2024-04-22T01:12:17.463203+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:12:17.463203+0000","last_undegraded":"2024-04-22T01:12:17.463203+0000","last_fullsized":"2024-04-22T01:12:17.463203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:43.721209+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:12:43.721209+0000","last_peered":"2024-04-22T01:12:43.721209+0000","last_clean":"2024-04-22T01:12:43.721209+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:12:43.721209+0000","last_undegraded":"2024-04-22T01:12:43.721209+0000","last_fullsized":"2024-04-22T01:12:43.721209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.791049+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:12:17.791049+0000","last_peered":"2024-04-22T01:12:17.791049+0000","last_clean":"2024-04-22T01:12:17.791049+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:12:17.791049+0000","last_undegraded":"2024-04-22T01:12:17.791049+0000","last_fullsized":"2024-04-22T01:12:17.791049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467241+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:12:17.467241+0000","last_peered":"2024-04-22T01:12:17.467241+0000","last_clean":"2024-04-22T01:12:17.467241+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:12:17.467241+0000","last_undegraded":"2024-04-22T01:12:17.467241+0000","last_fullsized":"2024-04-22T01:12:17.467241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.792676+0000","last_change":"2024-04-22T01:12:17.791976+0000","last_active":"2024-04-22T01:12:17.792676+0000","last_peered":"2024-04-22T01:12:17.792676+0000","last_clean":"2024-04-22T01:12:17.792676+0000","last_became_active":"2024-04-22T01:12:17.791528+0000","last_became_peered":"2024-04-22T01:12:17.791528+0000","last_unstale":"2024-04-22T01:12:17.792676+0000","last_undegraded":"2024-04-22T01:12:17.792676+0000","last_fullsized":"2024-04-22T01:12:17.792676+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:24.398418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466197+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:12:17.466197+0000","last_peered":"2024-04-22T01:12:17.466197+0000","last_clean":"2024-04-22T01:12:17.466197+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:12:17.466197+0000","last_undegraded":"2024-04-22T01:12:17.466197+0000","last_fullsized":"2024-04-22T01:12:17.466197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462078+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:12:17.462078+0000","last_peered":"2024-04-22T01:12:17.462078+0000","last_clean":"2024-04-22T01:12:17.462078+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:12:17.462078+0000","last_undegraded":"2024-04-22T01:12:17.462078+0000","last_fullsized":"2024-04-22T01:12:17.462078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.531275+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:13:00.531275+0000","last_peered":"2024-04-22T01:13:00.531275+0000","last_clean":"2024-04-22T01:13:00.531275+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:13:00.531275+0000","last_undegraded":"2024-04-22T01:13:00.531275+0000","last_fullsized":"2024-04-22T01:13:00.531275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465714+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:12:17.465714+0000","last_peered":"2024-04-22T01:12:17.465714+0000","last_clean":"2024-04-22T01:12:17.465714+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:12:17.465714+0000","last_undegraded":"2024-04-22T01:12:17.465714+0000","last_fullsized":"2024-04-22T01:12:17.465714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462218+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:12:17.462218+0000","last_peered":"2024-04-22T01:12:17.462218+0000","last_clean":"2024-04-22T01:12:17.462218+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:12:17.462218+0000","last_undegraded":"2024-04-22T01:12:17.462218+0000","last_fullsized":"2024-04-22T01:12:17.462218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462941+0000","last_change":"2024-04-22T01:11:17.586033+0000","last_active":"2024-04-22T01:12:17.462941+0000","last_peered":"2024-04-22T01:12:17.462941+0000","last_clean":"2024-04-22T01:12:17.462941+0000","last_became_active":"2024-04-22T01:11:17.585529+0000","last_became_peered":"2024-04-22T01:11:17.585529+0000","last_unstale":"2024-04-22T01:12:17.462941+0000","last_undegraded":"2024-04-22T01:12:17.462941+0000","last_fullsized":"2024-04-22T01:12:17.462941+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:35:49.219761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:01.417087+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:13:01.417087+0000","last_peered":"2024-04-22T01:13:01.417087+0000","last_clean":"2024-04-22T01:13:01.417087+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:13:01.417087+0000","last_undegraded":"2024-04-22T01:13:01.417087+0000","last_fullsized":"2024-04-22T01:13:01.417087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468114+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:12:17.468114+0000","last_peered":"2024-04-22T01:12:17.468114+0000","last_clean":"2024-04-22T01:12:17.468114+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:12:17.468114+0000","last_undegraded":"2024-04-22T01:12:17.468114+0000","last_fullsized":"2024-04-22T01:12:17.468114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467519+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:12:17.467519+0000","last_peered":"2024-04-22T01:12:17.467519+0000","last_clean":"2024-04-22T01:12:17.467519+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:12:17.467519+0000","last_undegraded":"2024-04-22T01:12:17.467519+0000","last_fullsized":"2024-04-22T01:12:17.467519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464425+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:12:17.464425+0000","last_peered":"2024-04-22T01:12:17.464425+0000","last_clean":"2024-04-22T01:12:17.464425+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:12:17.464425+0000","last_undegraded":"2024-04-22T01:12:17.464425+0000","last_fullsized":"2024-04-22T01:12:17.464425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467202+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:12:17.467202+0000","last_peered":"2024-04-22T01:12:17.467202+0000","last_clean":"2024-04-22T01:12:17.467202+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:12:17.467202+0000","last_undegraded":"2024-04-22T01:12:17.467202+0000","last_fullsized":"2024-04-22T01:12:17.467202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463203+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:12:17.463203+0000","last_peered":"2024-04-22T01:12:17.463203+0000","last_clean":"2024-04-22T01:12:17.463203+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:12:17.463203+0000","last_undegraded":"2024-04-22T01:12:17.463203+0000","last_fullsized":"2024-04-22T01:12:17.463203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.461817+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:12:17.461817+0000","last_peered":"2024-04-22T01:12:17.461817+0000","last_clean":"2024-04-22T01:12:17.461817+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:12:17.461817+0000","last_undegraded":"2024-04-22T01:12:17.461817+0000","last_fullsized":"2024-04-22T01:12:17.461817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467945+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:12:17.467945+0000","last_peered":"2024-04-22T01:12:17.467945+0000","last_clean":"2024-04-22T01:12:17.467945+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:12:17.467945+0000","last_undegraded":"2024-04-22T01:12:17.467945+0000","last_fullsized":"2024-04-22T01:12:17.467945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790799+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:12:17.790799+0000","last_peered":"2024-04-22T01:12:17.790799+0000","last_clean":"2024-04-22T01:12:17.790799+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:12:17.790799+0000","last_undegraded":"2024-04-22T01:12:17.790799+0000","last_fullsized":"2024-04-22T01:12:17.790799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467511+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:12:17.467511+0000","last_peered":"2024-04-22T01:12:17.467511+0000","last_clean":"2024-04-22T01:12:17.467511+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:12:17.467511+0000","last_undegraded":"2024-04-22T01:12:17.467511+0000","last_fullsized":"2024-04-22T01:12:17.467511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462204+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:12:17.462204+0000","last_peered":"2024-04-22T01:12:17.462204+0000","last_clean":"2024-04-22T01:12:17.462204+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:12:17.462204+0000","last_undegraded":"2024-04-22T01:12:17.462204+0000","last_fullsized":"2024-04-22T01:12:17.462204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467391+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:12:17.467391+0000","last_peered":"2024-04-22T01:12:17.467391+0000","last_clean":"2024-04-22T01:12:17.467391+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:12:17.467391+0000","last_undegraded":"2024-04-22T01:12:17.467391+0000","last_fullsized":"2024-04-22T01:12:17.467391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.797430+0000","last_change":"2024-04-22T01:12:17.796919+0000","last_active":"2024-04-22T01:12:17.797430+0000","last_peered":"2024-04-22T01:12:17.797430+0000","last_clean":"2024-04-22T01:12:17.797430+0000","last_became_active":"2024-04-22T01:12:17.791725+0000","last_became_peered":"2024-04-22T01:12:17.791725+0000","last_unstale":"2024-04-22T01:12:17.797430+0000","last_undegraded":"2024-04-22T01:12:17.797430+0000","last_fullsized":"2024-04-22T01:12:17.797430+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:50.607134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468273+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:12:17.468273+0000","last_peered":"2024-04-22T01:12:17.468273+0000","last_clean":"2024-04-22T01:12:17.468273+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:12:17.468273+0000","last_undegraded":"2024-04-22T01:12:17.468273+0000","last_fullsized":"2024-04-22T01:12:17.468273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.469923+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:12:17.469923+0000","last_peered":"2024-04-22T01:12:17.469923+0000","last_clean":"2024-04-22T01:12:17.469923+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:12:17.469923+0000","last_undegraded":"2024-04-22T01:12:17.469923+0000","last_fullsized":"2024-04-22T01:12:17.469923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465115+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:12:17.465115+0000","last_peered":"2024-04-22T01:12:17.465115+0000","last_clean":"2024-04-22T01:12:17.465115+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:12:17.465115+0000","last_undegraded":"2024-04-22T01:12:17.465115+0000","last_fullsized":"2024-04-22T01:12:17.465115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468452+0000","last_change":"2024-04-22T01:11:17.524897+0000","last_active":"2024-04-22T01:12:17.468452+0000","last_peered":"2024-04-22T01:12:17.468452+0000","last_clean":"2024-04-22T01:12:17.468452+0000","last_became_active":"2024-04-22T01:11:17.524423+0000","last_became_peered":"2024-04-22T01:11:17.524423+0000","last_unstale":"2024-04-22T01:12:17.468452+0000","last_undegraded":"2024-04-22T01:12:17.468452+0000","last_fullsized":"2024-04-22T01:12:17.468452+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:15.247122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464754+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:12:17.464754+0000","last_peered":"2024-04-22T01:12:17.464754+0000","last_clean":"2024-04-22T01:12:17.464754+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:12:17.464754+0000","last_undegraded":"2024-04-22T01:12:17.464754+0000","last_fullsized":"2024-04-22T01:12:17.464754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470357+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:12:17.470357+0000","last_peered":"2024-04-22T01:12:17.470357+0000","last_clean":"2024-04-22T01:12:17.470357+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:12:17.470357+0000","last_undegraded":"2024-04-22T01:12:17.470357+0000","last_fullsized":"2024-04-22T01:12:17.470357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467064+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:12:17.467064+0000","last_peered":"2024-04-22T01:12:17.467064+0000","last_clean":"2024-04-22T01:12:17.467064+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:12:17.467064+0000","last_undegraded":"2024-04-22T01:12:17.467064+0000","last_fullsized":"2024-04-22T01:12:17.467064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471987+0000","last_change":"2024-04-22T01:12:17.471987+0000","last_active":"2024-04-22T01:12:17.471987+0000","last_peered":"2024-04-22T01:12:17.471987+0000","last_clean":"2024-04-22T01:12:17.471987+0000","last_became_active":"2024-04-22T01:12:17.471369+0000","last_became_peered":"2024-04-22T01:12:17.471369+0000","last_unstale":"2024-04-22T01:12:17.471987+0000","last_undegraded":"2024-04-22T01:12:17.471987+0000","last_fullsized":"2024-04-22T01:12:17.471987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:20.856968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,8],"acting":[10,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467697+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:12:17.467697+0000","last_peered":"2024-04-22T01:12:17.467697+0000","last_clean":"2024-04-22T01:12:17.467697+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:12:17.467697+0000","last_undegraded":"2024-04-22T01:12:17.467697+0000","last_fullsized":"2024-04-22T01:12:17.467697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790341+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:12:17.790341+0000","last_peered":"2024-04-22T01:12:17.790341+0000","last_clean":"2024-04-22T01:12:17.790341+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:12:17.790341+0000","last_undegraded":"2024-04-22T01:12:17.790341+0000","last_fullsized":"2024-04-22T01:12:17.790341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464549+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:12:17.464549+0000","last_peered":"2024-04-22T01:12:17.464549+0000","last_clean":"2024-04-22T01:12:17.464549+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:12:17.464549+0000","last_undegraded":"2024-04-22T01:12:17.464549+0000","last_fullsized":"2024-04-22T01:12:17.464549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:38.991705+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:12:38.991705+0000","last_peered":"2024-04-22T01:12:38.991705+0000","last_clean":"2024-04-22T01:12:38.991705+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:12:38.991705+0000","last_undegraded":"2024-04-22T01:12:38.991705+0000","last_fullsized":"2024-04-22T01:12:38.991705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467870+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:12:17.467870+0000","last_peered":"2024-04-22T01:12:17.467870+0000","last_clean":"2024-04-22T01:12:17.467870+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:12:17.467870+0000","last_undegraded":"2024-04-22T01:12:17.467870+0000","last_fullsized":"2024-04-22T01:12:17.467870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464696+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:12:17.464696+0000","last_peered":"2024-04-22T01:12:17.464696+0000","last_clean":"2024-04-22T01:12:17.464696+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:12:17.464696+0000","last_undegraded":"2024-04-22T01:12:17.464696+0000","last_fullsized":"2024-04-22T01:12:17.464696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467236+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:12:17.467236+0000","last_peered":"2024-04-22T01:12:17.467236+0000","last_clean":"2024-04-22T01:12:17.467236+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:12:17.467236+0000","last_undegraded":"2024-04-22T01:12:17.467236+0000","last_fullsized":"2024-04-22T01:12:17.467236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468077+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:12:17.468077+0000","last_peered":"2024-04-22T01:12:17.468077+0000","last_clean":"2024-04-22T01:12:17.468077+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:12:17.468077+0000","last_undegraded":"2024-04-22T01:12:17.468077+0000","last_fullsized":"2024-04-22T01:12:17.468077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470610+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:12:17.470610+0000","last_peered":"2024-04-22T01:12:17.470610+0000","last_clean":"2024-04-22T01:12:17.470610+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:12:17.470610+0000","last_undegraded":"2024-04-22T01:12:17.470610+0000","last_fullsized":"2024-04-22T01:12:17.470610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:03.624385+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:13:03.624385+0000","last_peered":"2024-04-22T01:13:03.624385+0000","last_clean":"2024-04-22T01:13:03.624385+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:13:03.624385+0000","last_undegraded":"2024-04-22T01:13:03.624385+0000","last_fullsized":"2024-04-22T01:13:03.624385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2093,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468326+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:12:17.468326+0000","last_peered":"2024-04-22T01:12:17.468326+0000","last_clean":"2024-04-22T01:12:17.468326+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:12:17.468326+0000","last_undegraded":"2024-04-22T01:12:17.468326+0000","last_fullsized":"2024-04-22T01:12:17.468326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467471+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:12:17.467471+0000","last_peered":"2024-04-22T01:12:17.467471+0000","last_clean":"2024-04-22T01:12:17.467471+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:12:17.467471+0000","last_undegraded":"2024-04-22T01:12:17.467471+0000","last_fullsized":"2024-04-22T01:12:17.467471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463429+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:12:17.463429+0000","last_peered":"2024-04-22T01:12:17.463429+0000","last_clean":"2024-04-22T01:12:17.463429+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:12:17.463429+0000","last_undegraded":"2024-04-22T01:12:17.463429+0000","last_fullsized":"2024-04-22T01:12:17.463429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467776+0000","last_change":"2024-04-22T01:11:17.313900+0000","last_active":"2024-04-22T01:12:17.467776+0000","last_peered":"2024-04-22T01:12:17.467776+0000","last_clean":"2024-04-22T01:12:17.467776+0000","last_became_active":"2024-04-22T01:11:17.313493+0000","last_became_peered":"2024-04-22T01:11:17.313493+0000","last_unstale":"2024-04-22T01:12:17.467776+0000","last_undegraded":"2024-04-22T01:12:17.467776+0000","last_fullsized":"2024-04-22T01:12:17.467776+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:30:42.909373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462618+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:12:17.462618+0000","last_peered":"2024-04-22T01:12:17.462618+0000","last_clean":"2024-04-22T01:12:17.462618+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:12:17.462618+0000","last_undegraded":"2024-04-22T01:12:17.462618+0000","last_fullsized":"2024-04-22T01:12:17.462618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462388+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:12:17.462388+0000","last_peered":"2024-04-22T01:12:17.462388+0000","last_clean":"2024-04-22T01:12:17.462388+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:12:17.462388+0000","last_undegraded":"2024-04-22T01:12:17.462388+0000","last_fullsized":"2024-04-22T01:12:17.462388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468003+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:12:17.468003+0000","last_peered":"2024-04-22T01:12:17.468003+0000","last_clean":"2024-04-22T01:12:17.468003+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:12:17.468003+0000","last_undegraded":"2024-04-22T01:12:17.468003+0000","last_fullsized":"2024-04-22T01:12:17.468003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790421+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:12:17.790421+0000","last_peered":"2024-04-22T01:12:17.790421+0000","last_clean":"2024-04-22T01:12:17.790421+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:12:17.790421+0000","last_undegraded":"2024-04-22T01:12:17.790421+0000","last_fullsized":"2024-04-22T01:12:17.790421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470087+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:12:17.470087+0000","last_peered":"2024-04-22T01:12:17.470087+0000","last_clean":"2024-04-22T01:12:17.470087+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:12:17.470087+0000","last_undegraded":"2024-04-22T01:12:17.470087+0000","last_fullsized":"2024-04-22T01:12:17.470087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470224+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:12:17.470224+0000","last_peered":"2024-04-22T01:12:17.470224+0000","last_clean":"2024-04-22T01:12:17.470224+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:12:17.470224+0000","last_undegraded":"2024-04-22T01:12:17.470224+0000","last_fullsized":"2024-04-22T01:12:17.470224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467790+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:12:17.467790+0000","last_peered":"2024-04-22T01:12:17.467790+0000","last_clean":"2024-04-22T01:12:17.467790+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:12:17.467790+0000","last_undegraded":"2024-04-22T01:12:17.467790+0000","last_fullsized":"2024-04-22T01:12:17.467790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462189+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:12:17.462189+0000","last_peered":"2024-04-22T01:12:17.462189+0000","last_clean":"2024-04-22T01:12:17.462189+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:12:17.462189+0000","last_undegraded":"2024-04-22T01:12:17.462189+0000","last_fullsized":"2024-04-22T01:12:17.462189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790840+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:12:17.790840+0000","last_peered":"2024-04-22T01:12:17.790840+0000","last_clean":"2024-04-22T01:12:17.790840+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:12:17.790840+0000","last_undegraded":"2024-04-22T01:12:17.790840+0000","last_fullsized":"2024-04-22T01:12:17.790840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470692+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:12:17.470692+0000","last_peered":"2024-04-22T01:12:17.470692+0000","last_clean":"2024-04-22T01:12:17.470692+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:12:17.470692+0000","last_undegraded":"2024-04-22T01:12:17.470692+0000","last_fullsized":"2024-04-22T01:12:17.470692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468527+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:12:17.468527+0000","last_peered":"2024-04-22T01:12:17.468527+0000","last_clean":"2024-04-22T01:12:17.468527+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:12:17.468527+0000","last_undegraded":"2024-04-22T01:12:17.468527+0000","last_fullsized":"2024-04-22T01:12:17.468527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462784+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:12:17.462784+0000","last_peered":"2024-04-22T01:12:17.462784+0000","last_clean":"2024-04-22T01:12:17.462784+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:12:17.462784+0000","last_undegraded":"2024-04-22T01:12:17.462784+0000","last_fullsized":"2024-04-22T01:12:17.462784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467872+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:12:17.467872+0000","last_peered":"2024-04-22T01:12:17.467872+0000","last_clean":"2024-04-22T01:12:17.467872+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:12:17.467872+0000","last_undegraded":"2024-04-22T01:12:17.467872+0000","last_fullsized":"2024-04-22T01:12:17.467872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470799+0000","last_change":"2024-04-22T01:11:17.586313+0000","last_active":"2024-04-22T01:12:17.470799+0000","last_peered":"2024-04-22T01:12:17.470799+0000","last_clean":"2024-04-22T01:12:17.470799+0000","last_became_active":"2024-04-22T01:11:17.585705+0000","last_became_peered":"2024-04-22T01:11:17.585705+0000","last_unstale":"2024-04-22T01:12:17.470799+0000","last_undegraded":"2024-04-22T01:12:17.470799+0000","last_fullsized":"2024-04-22T01:12:17.470799+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:02.313774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462485+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:12:17.462485+0000","last_peered":"2024-04-22T01:12:17.462485+0000","last_clean":"2024-04-22T01:12:17.462485+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:12:17.462485+0000","last_undegraded":"2024-04-22T01:12:17.462485+0000","last_fullsized":"2024-04-22T01:12:17.462485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790617+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:12:17.790617+0000","last_peered":"2024-04-22T01:12:17.790617+0000","last_clean":"2024-04-22T01:12:17.790617+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:12:17.790617+0000","last_undegraded":"2024-04-22T01:12:17.790617+0000","last_fullsized":"2024-04-22T01:12:17.790617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69775,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1256,"num_read_kb":700,"num_write":257,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":672,"num_keys_recovered":7,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":217953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":521373,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1175552,"data_stored":1168157,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":524476,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":520086,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1117704,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495687,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":515884,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":583490,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":517872,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":584884,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":515592,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":519958,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1113369,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:13:06.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:13:07.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:06 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3210968870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:06 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1738362109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:06 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2278702210' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:07.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:06 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:13:07.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 -- 172.21.15.38:0/1625382645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 msgr2=0x7f1370073b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 --2- 172.21.15.38:0/1625382645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370073b50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1360009900 tx=0x7f136002bcd0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 -- 172.21.15.38:0/1625382645 shutdown_connections 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 --2- 172.21.15.38:0/1625382645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1370109640 0x7f137010bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 --2- 172.21.15.38:0/1625382645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370074120 0x7f1370074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 --2- 172.21.15.38:0/1625382645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370073b50 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 -- 172.21.15.38:0/1625382645 >> 172.21.15.38:0/1625382645 conn(0x7f13700fcd20 msgr2=0x7f13700ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 -- 172.21.15.38:0/1625382645 shutdown_connections 2024-04-22T01:13:07.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.090+0000 7f1375719640 1 -- 172.21.15.38:0/1625382645 wait complete. 2024-04-22T01:13:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 Processor -- start 2024-04-22T01:13:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 -- start start 2024-04-22T01:13:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1370074120 0x7f1370104b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370105070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370109640 0x7f13701016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f137010f640 con 0x7f1370075700 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f137010f4c0 con 0x7f1370074120 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f1375719640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f137010f7c0 con 0x7f1370109640 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f136e7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370105070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f136e7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370105070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37302/0 (socket says 172.21.15.38:37302) 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f136f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370109640 0x7f13701016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f136e7fc640 1 -- 172.21.15.38:0/3407983118 learned_addr learned my addr 172.21.15.38:0/3407983118 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.091+0000 7f136effd640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1370074120 0x7f1370104b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136f7fe640 1 -- 172.21.15.38:0/3407983118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1370074120 msgr2=0x7f1370104b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136f7fe640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1370074120 0x7f1370104b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136f7fe640 1 -- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 msgr2=0x7f1370105070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136f7fe640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370105070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136f7fe640 1 -- 172.21.15.38:0/3407983118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1370101f60 con 0x7f1370109640 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136e7fc640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f136f7fe640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370109640 0x7f13701016a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1368009fd0 tx=0x7f136800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136800ce20 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13701a60e0 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1368005020 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1368010720 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.092+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13701a64f0 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.093+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1370108400 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.094+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f1368005540 con 0x7f1370109640 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.095+0000 7f1357fff640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f13580a2c20 0x7f13580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.095+0000 7f136effd640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f13580a2c20 0x7f13580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.095+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f13680cfc10 con 0x7f1370109640 2024-04-22T01:13:07.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.096+0000 7f136effd640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f13580a2c20 0x7f13580a50e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f136002c2e0 tx=0x7f136002c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:07.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.099+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f136809c640 con 0x7f1370109640 2024-04-22T01:13:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:06 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3210968870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:06 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1738362109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:06 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2278702210' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:07.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:06 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:13:07.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.298+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1370061b80 con 0x7f13580a2c20 2024-04-22T01:13:07.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.307+0000 7f1357fff640 1 -- 172.21.15.38:0/3407983118 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340810 (secure 0 0 0) 0x7f1370061b80 con 0x7f13580a2c20 2024-04-22T01:13:07.310 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:07.312 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f13580a2c20 msgr2=0x7f13580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f13580a2c20 0x7f13580a50e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f136002c2e0 tx=0x7f136002c920 comp rx=0 tx=0).stop 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370109640 msgr2=0x7f13701016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370109640 0x7f13701016a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1368009fd0 tx=0x7f136800ef10 comp rx=0 tx=0).stop 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 shutdown_connections 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f13580a2c20 0x7f13580a50e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1370109640 0x7f13701016a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1370075700 0x7f1370105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 --2- 172.21.15.38:0/3407983118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1370074120 0x7f1370104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 >> 172.21.15.38:0/3407983118 conn(0x7f13700fcd20 msgr2=0x7f13701055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:07.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 shutdown_connections 2024-04-22T01:13:07.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.311+0000 7f1375719640 1 -- 172.21.15.38:0/3407983118 wait complete. 2024-04-22T01:13:07.323 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-04-22T01:13:05.826500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660377,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1382,"num_read_kb":809,"num_write":440,"num_write_kb":3091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181408,"num_keys_recovered":7,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357636,"kb_used_data":12592,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584188,"statfs":{"total":1151940427776,"available":1151574208512,"internally_reserved":0,"allocated":12894208,"data_stored":8202845,"data_compressed":428511,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1398,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":52,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790906+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:12:17.790906+0000","last_peered":"2024-04-22T01:12:17.790906+0000","last_clean":"2024-04-22T01:12:17.790906+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:12:17.790906+0000","last_undegraded":"2024-04-22T01:12:17.790906+0000","last_fullsized":"2024-04-22T01:12:17.790906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468337+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:12:17.468337+0000","last_peered":"2024-04-22T01:12:17.468337+0000","last_clean":"2024-04-22T01:12:17.468337+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:12:17.468337+0000","last_undegraded":"2024-04-22T01:12:17.468337+0000","last_fullsized":"2024-04-22T01:12:17.468337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467910+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:12:17.467910+0000","last_peered":"2024-04-22T01:12:17.467910+0000","last_clean":"2024-04-22T01:12:17.467910+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:12:17.467910+0000","last_undegraded":"2024-04-22T01:12:17.467910+0000","last_fullsized":"2024-04-22T01:12:17.467910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464881+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:12:17.464881+0000","last_peered":"2024-04-22T01:12:17.464881+0000","last_clean":"2024-04-22T01:12:17.464881+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:12:17.464881+0000","last_undegraded":"2024-04-22T01:12:17.464881+0000","last_fullsized":"2024-04-22T01:12:17.464881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467393+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:12:17.467393+0000","last_peered":"2024-04-22T01:12:17.467393+0000","last_clean":"2024-04-22T01:12:17.467393+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:12:17.467393+0000","last_undegraded":"2024-04-22T01:12:17.467393+0000","last_fullsized":"2024-04-22T01:12:17.467393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466033+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:12:17.466033+0000","last_peered":"2024-04-22T01:12:17.466033+0000","last_clean":"2024-04-22T01:12:17.466033+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:12:17.466033+0000","last_undegraded":"2024-04-22T01:12:17.466033+0000","last_fullsized":"2024-04-22T01:12:17.466033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467329+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:12:17.467329+0000","last_peered":"2024-04-22T01:12:17.467329+0000","last_clean":"2024-04-22T01:12:17.467329+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:12:17.467329+0000","last_undegraded":"2024-04-22T01:12:17.467329+0000","last_fullsized":"2024-04-22T01:12:17.467329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467174+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:12:17.467174+0000","last_peered":"2024-04-22T01:12:17.467174+0000","last_clean":"2024-04-22T01:12:17.467174+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:12:17.467174+0000","last_undegraded":"2024-04-22T01:12:17.467174+0000","last_fullsized":"2024-04-22T01:12:17.467174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471434+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:12:17.471434+0000","last_peered":"2024-04-22T01:12:17.471434+0000","last_clean":"2024-04-22T01:12:17.471434+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:12:17.471434+0000","last_undegraded":"2024-04-22T01:12:17.471434+0000","last_fullsized":"2024-04-22T01:12:17.471434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466554+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:12:17.466554+0000","last_peered":"2024-04-22T01:12:17.466554+0000","last_clean":"2024-04-22T01:12:17.466554+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:12:17.466554+0000","last_undegraded":"2024-04-22T01:12:17.466554+0000","last_fullsized":"2024-04-22T01:12:17.466554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468253+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:12:17.468253+0000","last_peered":"2024-04-22T01:12:17.468253+0000","last_clean":"2024-04-22T01:12:17.468253+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:12:17.468253+0000","last_undegraded":"2024-04-22T01:12:17.468253+0000","last_fullsized":"2024-04-22T01:12:17.468253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464762+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:12:17.464762+0000","last_peered":"2024-04-22T01:12:17.464762+0000","last_clean":"2024-04-22T01:12:17.464762+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:12:17.464762+0000","last_undegraded":"2024-04-22T01:12:17.464762+0000","last_fullsized":"2024-04-22T01:12:17.464762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462419+0000","last_change":"2024-04-22T01:11:17.312524+0000","last_active":"2024-04-22T01:12:17.462419+0000","last_peered":"2024-04-22T01:12:17.462419+0000","last_clean":"2024-04-22T01:12:17.462419+0000","last_became_active":"2024-04-22T01:11:17.312068+0000","last_became_peered":"2024-04-22T01:11:17.312068+0000","last_unstale":"2024-04-22T01:12:17.462419+0000","last_undegraded":"2024-04-22T01:12:17.462419+0000","last_fullsized":"2024-04-22T01:12:17.462419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:47:17.243712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467343+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:12:17.467343+0000","last_peered":"2024-04-22T01:12:17.467343+0000","last_clean":"2024-04-22T01:12:17.467343+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:12:17.467343+0000","last_undegraded":"2024-04-22T01:12:17.467343+0000","last_fullsized":"2024-04-22T01:12:17.467343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467849+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:12:17.467849+0000","last_peered":"2024-04-22T01:12:17.467849+0000","last_clean":"2024-04-22T01:12:17.467849+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:12:17.467849+0000","last_undegraded":"2024-04-22T01:12:17.467849+0000","last_fullsized":"2024-04-22T01:12:17.467849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470498+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:12:17.470498+0000","last_peered":"2024-04-22T01:12:17.470498+0000","last_clean":"2024-04-22T01:12:17.470498+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:12:17.470498+0000","last_undegraded":"2024-04-22T01:12:17.470498+0000","last_fullsized":"2024-04-22T01:12:17.470498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790752+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:12:17.790752+0000","last_peered":"2024-04-22T01:12:17.790752+0000","last_clean":"2024-04-22T01:12:17.790752+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:12:17.790752+0000","last_undegraded":"2024-04-22T01:12:17.790752+0000","last_fullsized":"2024-04-22T01:12:17.790752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465060+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:12:17.465060+0000","last_peered":"2024-04-22T01:12:17.465060+0000","last_clean":"2024-04-22T01:12:17.465060+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:12:17.465060+0000","last_undegraded":"2024-04-22T01:12:17.465060+0000","last_fullsized":"2024-04-22T01:12:17.465060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470457+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:12:17.470457+0000","last_peered":"2024-04-22T01:12:17.470457+0000","last_clean":"2024-04-22T01:12:17.470457+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:12:17.470457+0000","last_undegraded":"2024-04-22T01:12:17.470457+0000","last_fullsized":"2024-04-22T01:12:17.470457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470430+0000","last_change":"2024-04-22T01:11:17.586121+0000","last_active":"2024-04-22T01:12:17.470430+0000","last_peered":"2024-04-22T01:12:17.470430+0000","last_clean":"2024-04-22T01:12:17.470430+0000","last_became_active":"2024-04-22T01:11:17.585619+0000","last_became_peered":"2024-04-22T01:11:17.585619+0000","last_unstale":"2024-04-22T01:12:17.470430+0000","last_undegraded":"2024-04-22T01:12:17.470430+0000","last_fullsized":"2024-04-22T01:12:17.470430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:12.987810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:59.529087+0000","last_change":"2024-04-22T01:12:17.484778+0000","last_active":"2024-04-22T01:12:59.529087+0000","last_peered":"2024-04-22T01:12:59.529087+0000","last_clean":"2024-04-22T01:12:59.529087+0000","last_became_active":"2024-04-22T01:12:17.472811+0000","last_became_peered":"2024-04-22T01:12:17.472811+0000","last_unstale":"2024-04-22T01:12:59.529087+0000","last_undegraded":"2024-04-22T01:12:59.529087+0000","last_fullsized":"2024-04-22T01:12:59.529087+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:07.118642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467391+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:12:17.467391+0000","last_peered":"2024-04-22T01:12:17.467391+0000","last_clean":"2024-04-22T01:12:17.467391+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:12:17.467391+0000","last_undegraded":"2024-04-22T01:12:17.467391+0000","last_fullsized":"2024-04-22T01:12:17.467391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464207+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:12:17.464207+0000","last_peered":"2024-04-22T01:12:17.464207+0000","last_clean":"2024-04-22T01:12:17.464207+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:12:17.464207+0000","last_undegraded":"2024-04-22T01:12:17.464207+0000","last_fullsized":"2024-04-22T01:12:17.464207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468644+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:12:17.468644+0000","last_peered":"2024-04-22T01:12:17.468644+0000","last_clean":"2024-04-22T01:12:17.468644+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:12:17.468644+0000","last_undegraded":"2024-04-22T01:12:17.468644+0000","last_fullsized":"2024-04-22T01:12:17.468644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468490+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:12:17.468490+0000","last_peered":"2024-04-22T01:12:17.468490+0000","last_clean":"2024-04-22T01:12:17.468490+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:12:17.468490+0000","last_undegraded":"2024-04-22T01:12:17.468490+0000","last_fullsized":"2024-04-22T01:12:17.468490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464299+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:12:17.464299+0000","last_peered":"2024-04-22T01:12:17.464299+0000","last_clean":"2024-04-22T01:12:17.464299+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:12:17.464299+0000","last_undegraded":"2024-04-22T01:12:17.464299+0000","last_fullsized":"2024-04-22T01:12:17.464299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467663+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:12:17.467663+0000","last_peered":"2024-04-22T01:12:17.467663+0000","last_clean":"2024-04-22T01:12:17.467663+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:12:17.467663+0000","last_undegraded":"2024-04-22T01:12:17.467663+0000","last_fullsized":"2024-04-22T01:12:17.467663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470754+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:12:17.470754+0000","last_peered":"2024-04-22T01:12:17.470754+0000","last_clean":"2024-04-22T01:12:17.470754+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:12:17.470754+0000","last_undegraded":"2024-04-22T01:12:17.470754+0000","last_fullsized":"2024-04-22T01:12:17.470754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464253+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:12:17.464253+0000","last_peered":"2024-04-22T01:12:17.464253+0000","last_clean":"2024-04-22T01:12:17.464253+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:12:17.464253+0000","last_undegraded":"2024-04-22T01:12:17.464253+0000","last_fullsized":"2024-04-22T01:12:17.464253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467272+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:12:17.467272+0000","last_peered":"2024-04-22T01:12:17.467272+0000","last_clean":"2024-04-22T01:12:17.467272+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:12:17.467272+0000","last_undegraded":"2024-04-22T01:12:17.467272+0000","last_fullsized":"2024-04-22T01:12:17.467272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463194+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:12:17.463194+0000","last_peered":"2024-04-22T01:12:17.463194+0000","last_clean":"2024-04-22T01:12:17.463194+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:12:17.463194+0000","last_undegraded":"2024-04-22T01:12:17.463194+0000","last_fullsized":"2024-04-22T01:12:17.463194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470258+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:12:17.470258+0000","last_peered":"2024-04-22T01:12:17.470258+0000","last_clean":"2024-04-22T01:12:17.470258+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:12:17.470258+0000","last_undegraded":"2024-04-22T01:12:17.470258+0000","last_fullsized":"2024-04-22T01:12:17.470258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:03.625138+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:13:03.625138+0000","last_peered":"2024-04-22T01:13:03.625138+0000","last_clean":"2024-04-22T01:13:03.625138+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:13:03.625138+0000","last_undegraded":"2024-04-22T01:13:03.625138+0000","last_fullsized":"2024-04-22T01:13:03.625138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466990+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:12:17.466990+0000","last_peered":"2024-04-22T01:12:17.466990+0000","last_clean":"2024-04-22T01:12:17.466990+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:12:17.466990+0000","last_undegraded":"2024-04-22T01:12:17.466990+0000","last_fullsized":"2024-04-22T01:12:17.466990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470301+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:12:17.470301+0000","last_peered":"2024-04-22T01:12:17.470301+0000","last_clean":"2024-04-22T01:12:17.470301+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:12:17.470301+0000","last_undegraded":"2024-04-22T01:12:17.470301+0000","last_fullsized":"2024-04-22T01:12:17.470301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465883+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:12:17.465883+0000","last_peered":"2024-04-22T01:12:17.465883+0000","last_clean":"2024-04-22T01:12:17.465883+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:12:17.465883+0000","last_undegraded":"2024-04-22T01:12:17.465883+0000","last_fullsized":"2024-04-22T01:12:17.465883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463275+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:12:17.463275+0000","last_peered":"2024-04-22T01:12:17.463275+0000","last_clean":"2024-04-22T01:12:17.463275+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:12:17.463275+0000","last_undegraded":"2024-04-22T01:12:17.463275+0000","last_fullsized":"2024-04-22T01:12:17.463275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790698+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:12:17.790698+0000","last_peered":"2024-04-22T01:12:17.790698+0000","last_clean":"2024-04-22T01:12:17.790698+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:12:17.790698+0000","last_undegraded":"2024-04-22T01:12:17.790698+0000","last_fullsized":"2024-04-22T01:12:17.790698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.461932+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:12:17.461932+0000","last_peered":"2024-04-22T01:12:17.461932+0000","last_clean":"2024-04-22T01:12:17.461932+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:12:17.461932+0000","last_undegraded":"2024-04-22T01:12:17.461932+0000","last_fullsized":"2024-04-22T01:12:17.461932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467441+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:12:17.467441+0000","last_peered":"2024-04-22T01:12:17.467441+0000","last_clean":"2024-04-22T01:12:17.467441+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:12:17.467441+0000","last_undegraded":"2024-04-22T01:12:17.467441+0000","last_fullsized":"2024-04-22T01:12:17.467441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468371+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:12:17.468371+0000","last_peered":"2024-04-22T01:12:17.468371+0000","last_clean":"2024-04-22T01:12:17.468371+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:12:17.468371+0000","last_undegraded":"2024-04-22T01:12:17.468371+0000","last_fullsized":"2024-04-22T01:12:17.468371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462747+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:12:17.462747+0000","last_peered":"2024-04-22T01:12:17.462747+0000","last_clean":"2024-04-22T01:12:17.462747+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:12:17.462747+0000","last_undegraded":"2024-04-22T01:12:17.462747+0000","last_fullsized":"2024-04-22T01:12:17.462747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464591+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:12:17.464591+0000","last_peered":"2024-04-22T01:12:17.464591+0000","last_clean":"2024-04-22T01:12:17.464591+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:12:17.464591+0000","last_undegraded":"2024-04-22T01:12:17.464591+0000","last_fullsized":"2024-04-22T01:12:17.464591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467587+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:12:17.467587+0000","last_peered":"2024-04-22T01:12:17.467587+0000","last_clean":"2024-04-22T01:12:17.467587+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:12:17.467587+0000","last_undegraded":"2024-04-22T01:12:17.467587+0000","last_fullsized":"2024-04-22T01:12:17.467587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":279,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.619054+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:13:00.619054+0000","last_peered":"2024-04-22T01:13:00.619054+0000","last_clean":"2024-04-22T01:13:00.619054+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:13:00.619054+0000","last_undegraded":"2024-04-22T01:13:00.619054+0000","last_fullsized":"2024-04-22T01:13:00.619054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471276+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:12:17.471276+0000","last_peered":"2024-04-22T01:12:17.471276+0000","last_clean":"2024-04-22T01:12:17.471276+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:12:17.471276+0000","last_undegraded":"2024-04-22T01:12:17.471276+0000","last_fullsized":"2024-04-22T01:12:17.471276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464653+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:12:17.464653+0000","last_peered":"2024-04-22T01:12:17.464653+0000","last_clean":"2024-04-22T01:12:17.464653+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:12:17.464653+0000","last_undegraded":"2024-04-22T01:12:17.464653+0000","last_fullsized":"2024-04-22T01:12:17.464653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.477748+0000","last_change":"2024-04-22T01:12:17.477108+0000","last_active":"2024-04-22T01:12:17.477748+0000","last_peered":"2024-04-22T01:12:17.477748+0000","last_clean":"2024-04-22T01:12:17.477748+0000","last_became_active":"2024-04-22T01:12:17.471865+0000","last_became_peered":"2024-04-22T01:12:17.471865+0000","last_unstale":"2024-04-22T01:12:17.477748+0000","last_undegraded":"2024-04-22T01:12:17.477748+0000","last_fullsized":"2024-04-22T01:12:17.477748+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:49:55.734197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468386+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:12:17.468386+0000","last_peered":"2024-04-22T01:12:17.468386+0000","last_clean":"2024-04-22T01:12:17.468386+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:12:17.468386+0000","last_undegraded":"2024-04-22T01:12:17.468386+0000","last_fullsized":"2024-04-22T01:12:17.468386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464504+0000","last_change":"2024-04-22T01:11:17.586221+0000","last_active":"2024-04-22T01:12:17.464504+0000","last_peered":"2024-04-22T01:12:17.464504+0000","last_clean":"2024-04-22T01:12:17.464504+0000","last_became_active":"2024-04-22T01:11:17.585689+0000","last_became_peered":"2024-04-22T01:11:17.585689+0000","last_unstale":"2024-04-22T01:12:17.464504+0000","last_undegraded":"2024-04-22T01:12:17.464504+0000","last_fullsized":"2024-04-22T01:12:17.464504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:40.327272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463285+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:12:17.463285+0000","last_peered":"2024-04-22T01:12:17.463285+0000","last_clean":"2024-04-22T01:12:17.463285+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:12:17.463285+0000","last_undegraded":"2024-04-22T01:12:17.463285+0000","last_fullsized":"2024-04-22T01:12:17.463285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470885+0000","last_change":"2024-04-22T01:11:17.585668+0000","last_active":"2024-04-22T01:12:17.470885+0000","last_peered":"2024-04-22T01:12:17.470885+0000","last_clean":"2024-04-22T01:12:17.470885+0000","last_became_active":"2024-04-22T01:11:17.585217+0000","last_became_peered":"2024-04-22T01:11:17.585217+0000","last_unstale":"2024-04-22T01:12:17.470885+0000","last_undegraded":"2024-04-22T01:12:17.470885+0000","last_fullsized":"2024-04-22T01:12:17.470885+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:24.007210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464709+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:12:17.464709+0000","last_peered":"2024-04-22T01:12:17.464709+0000","last_clean":"2024-04-22T01:12:17.464709+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:12:17.464709+0000","last_undegraded":"2024-04-22T01:12:17.464709+0000","last_fullsized":"2024-04-22T01:12:17.464709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'38","reported_seq":318,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:01.485731+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:13:01.485731+0000","last_peered":"2024-04-22T01:13:01.485731+0000","last_clean":"2024-04-22T01:13:01.485731+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:13:01.485731+0000","last_undegraded":"2024-04-22T01:13:01.485731+0000","last_fullsized":"2024-04-22T01:13:01.485731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61199,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":249,"num_read_kb":184,"num_write":40,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467503+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:12:17.467503+0000","last_peered":"2024-04-22T01:12:17.467503+0000","last_clean":"2024-04-22T01:12:17.467503+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:12:17.467503+0000","last_undegraded":"2024-04-22T01:12:17.467503+0000","last_fullsized":"2024-04-22T01:12:17.467503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463529+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:12:17.463529+0000","last_peered":"2024-04-22T01:12:17.463529+0000","last_clean":"2024-04-22T01:12:17.463529+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:12:17.463529+0000","last_undegraded":"2024-04-22T01:12:17.463529+0000","last_fullsized":"2024-04-22T01:12:17.463529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470394+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:12:17.470394+0000","last_peered":"2024-04-22T01:12:17.470394+0000","last_clean":"2024-04-22T01:12:17.470394+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:12:17.470394+0000","last_undegraded":"2024-04-22T01:12:17.470394+0000","last_fullsized":"2024-04-22T01:12:17.470394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462066+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:12:17.462066+0000","last_peered":"2024-04-22T01:12:17.462066+0000","last_clean":"2024-04-22T01:12:17.462066+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:12:17.462066+0000","last_undegraded":"2024-04-22T01:12:17.462066+0000","last_fullsized":"2024-04-22T01:12:17.462066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790993+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:12:17.790993+0000","last_peered":"2024-04-22T01:12:17.790993+0000","last_clean":"2024-04-22T01:12:17.790993+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:12:17.790993+0000","last_undegraded":"2024-04-22T01:12:17.790993+0000","last_fullsized":"2024-04-22T01:12:17.790993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463155+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:12:17.463155+0000","last_peered":"2024-04-22T01:12:17.463155+0000","last_clean":"2024-04-22T01:12:17.463155+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:12:17.463155+0000","last_undegraded":"2024-04-22T01:12:17.463155+0000","last_fullsized":"2024-04-22T01:12:17.463155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467829+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:12:17.467829+0000","last_peered":"2024-04-22T01:12:17.467829+0000","last_clean":"2024-04-22T01:12:17.467829+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:12:17.467829+0000","last_undegraded":"2024-04-22T01:12:17.467829+0000","last_fullsized":"2024-04-22T01:12:17.467829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464953+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:12:17.464953+0000","last_peered":"2024-04-22T01:12:17.464953+0000","last_clean":"2024-04-22T01:12:17.464953+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:12:17.464953+0000","last_undegraded":"2024-04-22T01:12:17.464953+0000","last_fullsized":"2024-04-22T01:12:17.464953+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.654949+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:13:00.654949+0000","last_peered":"2024-04-22T01:13:00.654949+0000","last_clean":"2024-04-22T01:13:00.654949+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:13:00.654949+0000","last_undegraded":"2024-04-22T01:13:00.654949+0000","last_fullsized":"2024-04-22T01:13:00.654949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467681+0000","last_change":"2024-04-22T01:11:17.314098+0000","last_active":"2024-04-22T01:12:17.467681+0000","last_peered":"2024-04-22T01:12:17.467681+0000","last_clean":"2024-04-22T01:12:17.467681+0000","last_became_active":"2024-04-22T01:11:17.313663+0000","last_became_peered":"2024-04-22T01:11:17.313663+0000","last_unstale":"2024-04-22T01:12:17.467681+0000","last_undegraded":"2024-04-22T01:12:17.467681+0000","last_fullsized":"2024-04-22T01:12:17.467681+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:20:03.826857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462323+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:12:17.462323+0000","last_peered":"2024-04-22T01:12:17.462323+0000","last_clean":"2024-04-22T01:12:17.462323+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:12:17.462323+0000","last_undegraded":"2024-04-22T01:12:17.462323+0000","last_fullsized":"2024-04-22T01:12:17.462323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466996+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:12:17.466996+0000","last_peered":"2024-04-22T01:12:17.466996+0000","last_clean":"2024-04-22T01:12:17.466996+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:12:17.466996+0000","last_undegraded":"2024-04-22T01:12:17.466996+0000","last_fullsized":"2024-04-22T01:12:17.466996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462862+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:12:17.462862+0000","last_peered":"2024-04-22T01:12:17.462862+0000","last_clean":"2024-04-22T01:12:17.462862+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:12:17.462862+0000","last_undegraded":"2024-04-22T01:12:17.462862+0000","last_fullsized":"2024-04-22T01:12:17.462862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463203+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:12:17.463203+0000","last_peered":"2024-04-22T01:12:17.463203+0000","last_clean":"2024-04-22T01:12:17.463203+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:12:17.463203+0000","last_undegraded":"2024-04-22T01:12:17.463203+0000","last_fullsized":"2024-04-22T01:12:17.463203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:43.721209+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:12:43.721209+0000","last_peered":"2024-04-22T01:12:43.721209+0000","last_clean":"2024-04-22T01:12:43.721209+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:12:43.721209+0000","last_undegraded":"2024-04-22T01:12:43.721209+0000","last_fullsized":"2024-04-22T01:12:43.721209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.791049+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:12:17.791049+0000","last_peered":"2024-04-22T01:12:17.791049+0000","last_clean":"2024-04-22T01:12:17.791049+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:12:17.791049+0000","last_undegraded":"2024-04-22T01:12:17.791049+0000","last_fullsized":"2024-04-22T01:12:17.791049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467241+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:12:17.467241+0000","last_peered":"2024-04-22T01:12:17.467241+0000","last_clean":"2024-04-22T01:12:17.467241+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:12:17.467241+0000","last_undegraded":"2024-04-22T01:12:17.467241+0000","last_fullsized":"2024-04-22T01:12:17.467241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.792676+0000","last_change":"2024-04-22T01:12:17.791976+0000","last_active":"2024-04-22T01:12:17.792676+0000","last_peered":"2024-04-22T01:12:17.792676+0000","last_clean":"2024-04-22T01:12:17.792676+0000","last_became_active":"2024-04-22T01:12:17.791528+0000","last_became_peered":"2024-04-22T01:12:17.791528+0000","last_unstale":"2024-04-22T01:12:17.792676+0000","last_undegraded":"2024-04-22T01:12:17.792676+0000","last_fullsized":"2024-04-22T01:12:17.792676+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:24.398418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466197+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:12:17.466197+0000","last_peered":"2024-04-22T01:12:17.466197+0000","last_clean":"2024-04-22T01:12:17.466197+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:12:17.466197+0000","last_undegraded":"2024-04-22T01:12:17.466197+0000","last_fullsized":"2024-04-22T01:12:17.466197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462078+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:12:17.462078+0000","last_peered":"2024-04-22T01:12:17.462078+0000","last_clean":"2024-04-22T01:12:17.462078+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:12:17.462078+0000","last_undegraded":"2024-04-22T01:12:17.462078+0000","last_fullsized":"2024-04-22T01:12:17.462078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.531275+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:13:00.531275+0000","last_peered":"2024-04-22T01:13:00.531275+0000","last_clean":"2024-04-22T01:13:00.531275+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:13:00.531275+0000","last_undegraded":"2024-04-22T01:13:00.531275+0000","last_fullsized":"2024-04-22T01:13:00.531275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465714+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:12:17.465714+0000","last_peered":"2024-04-22T01:12:17.465714+0000","last_clean":"2024-04-22T01:12:17.465714+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:12:17.465714+0000","last_undegraded":"2024-04-22T01:12:17.465714+0000","last_fullsized":"2024-04-22T01:12:17.465714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462218+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:12:17.462218+0000","last_peered":"2024-04-22T01:12:17.462218+0000","last_clean":"2024-04-22T01:12:17.462218+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:12:17.462218+0000","last_undegraded":"2024-04-22T01:12:17.462218+0000","last_fullsized":"2024-04-22T01:12:17.462218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462941+0000","last_change":"2024-04-22T01:11:17.586033+0000","last_active":"2024-04-22T01:12:17.462941+0000","last_peered":"2024-04-22T01:12:17.462941+0000","last_clean":"2024-04-22T01:12:17.462941+0000","last_became_active":"2024-04-22T01:11:17.585529+0000","last_became_peered":"2024-04-22T01:11:17.585529+0000","last_unstale":"2024-04-22T01:12:17.462941+0000","last_undegraded":"2024-04-22T01:12:17.462941+0000","last_fullsized":"2024-04-22T01:12:17.462941+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:35:49.219761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:01.417087+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:13:01.417087+0000","last_peered":"2024-04-22T01:13:01.417087+0000","last_clean":"2024-04-22T01:13:01.417087+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:13:01.417087+0000","last_undegraded":"2024-04-22T01:13:01.417087+0000","last_fullsized":"2024-04-22T01:13:01.417087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468114+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:12:17.468114+0000","last_peered":"2024-04-22T01:12:17.468114+0000","last_clean":"2024-04-22T01:12:17.468114+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:12:17.468114+0000","last_undegraded":"2024-04-22T01:12:17.468114+0000","last_fullsized":"2024-04-22T01:12:17.468114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467519+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:12:17.467519+0000","last_peered":"2024-04-22T01:12:17.467519+0000","last_clean":"2024-04-22T01:12:17.467519+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:12:17.467519+0000","last_undegraded":"2024-04-22T01:12:17.467519+0000","last_fullsized":"2024-04-22T01:12:17.467519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464425+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:12:17.464425+0000","last_peered":"2024-04-22T01:12:17.464425+0000","last_clean":"2024-04-22T01:12:17.464425+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:12:17.464425+0000","last_undegraded":"2024-04-22T01:12:17.464425+0000","last_fullsized":"2024-04-22T01:12:17.464425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467202+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:12:17.467202+0000","last_peered":"2024-04-22T01:12:17.467202+0000","last_clean":"2024-04-22T01:12:17.467202+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:12:17.467202+0000","last_undegraded":"2024-04-22T01:12:17.467202+0000","last_fullsized":"2024-04-22T01:12:17.467202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463203+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:12:17.463203+0000","last_peered":"2024-04-22T01:12:17.463203+0000","last_clean":"2024-04-22T01:12:17.463203+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:12:17.463203+0000","last_undegraded":"2024-04-22T01:12:17.463203+0000","last_fullsized":"2024-04-22T01:12:17.463203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.461817+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:12:17.461817+0000","last_peered":"2024-04-22T01:12:17.461817+0000","last_clean":"2024-04-22T01:12:17.461817+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:12:17.461817+0000","last_undegraded":"2024-04-22T01:12:17.461817+0000","last_fullsized":"2024-04-22T01:12:17.461817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467945+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:12:17.467945+0000","last_peered":"2024-04-22T01:12:17.467945+0000","last_clean":"2024-04-22T01:12:17.467945+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:12:17.467945+0000","last_undegraded":"2024-04-22T01:12:17.467945+0000","last_fullsized":"2024-04-22T01:12:17.467945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790799+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:12:17.790799+0000","last_peered":"2024-04-22T01:12:17.790799+0000","last_clean":"2024-04-22T01:12:17.790799+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:12:17.790799+0000","last_undegraded":"2024-04-22T01:12:17.790799+0000","last_fullsized":"2024-04-22T01:12:17.790799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467511+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:12:17.467511+0000","last_peered":"2024-04-22T01:12:17.467511+0000","last_clean":"2024-04-22T01:12:17.467511+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:12:17.467511+0000","last_undegraded":"2024-04-22T01:12:17.467511+0000","last_fullsized":"2024-04-22T01:12:17.467511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462204+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:12:17.462204+0000","last_peered":"2024-04-22T01:12:17.462204+0000","last_clean":"2024-04-22T01:12:17.462204+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:12:17.462204+0000","last_undegraded":"2024-04-22T01:12:17.462204+0000","last_fullsized":"2024-04-22T01:12:17.462204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467391+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:12:17.467391+0000","last_peered":"2024-04-22T01:12:17.467391+0000","last_clean":"2024-04-22T01:12:17.467391+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:12:17.467391+0000","last_undegraded":"2024-04-22T01:12:17.467391+0000","last_fullsized":"2024-04-22T01:12:17.467391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.797430+0000","last_change":"2024-04-22T01:12:17.796919+0000","last_active":"2024-04-22T01:12:17.797430+0000","last_peered":"2024-04-22T01:12:17.797430+0000","last_clean":"2024-04-22T01:12:17.797430+0000","last_became_active":"2024-04-22T01:12:17.791725+0000","last_became_peered":"2024-04-22T01:12:17.791725+0000","last_unstale":"2024-04-22T01:12:17.797430+0000","last_undegraded":"2024-04-22T01:12:17.797430+0000","last_fullsized":"2024-04-22T01:12:17.797430+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:50.607134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468273+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:12:17.468273+0000","last_peered":"2024-04-22T01:12:17.468273+0000","last_clean":"2024-04-22T01:12:17.468273+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:12:17.468273+0000","last_undegraded":"2024-04-22T01:12:17.468273+0000","last_fullsized":"2024-04-22T01:12:17.468273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.469923+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:12:17.469923+0000","last_peered":"2024-04-22T01:12:17.469923+0000","last_clean":"2024-04-22T01:12:17.469923+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:12:17.469923+0000","last_undegraded":"2024-04-22T01:12:17.469923+0000","last_fullsized":"2024-04-22T01:12:17.469923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465115+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:12:17.465115+0000","last_peered":"2024-04-22T01:12:17.465115+0000","last_clean":"2024-04-22T01:12:17.465115+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:12:17.465115+0000","last_undegraded":"2024-04-22T01:12:17.465115+0000","last_fullsized":"2024-04-22T01:12:17.465115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468452+0000","last_change":"2024-04-22T01:11:17.524897+0000","last_active":"2024-04-22T01:12:17.468452+0000","last_peered":"2024-04-22T01:12:17.468452+0000","last_clean":"2024-04-22T01:12:17.468452+0000","last_became_active":"2024-04-22T01:11:17.524423+0000","last_became_peered":"2024-04-22T01:11:17.524423+0000","last_unstale":"2024-04-22T01:12:17.468452+0000","last_undegraded":"2024-04-22T01:12:17.468452+0000","last_fullsized":"2024-04-22T01:12:17.468452+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:15.247122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464754+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:12:17.464754+0000","last_peered":"2024-04-22T01:12:17.464754+0000","last_clean":"2024-04-22T01:12:17.464754+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:12:17.464754+0000","last_undegraded":"2024-04-22T01:12:17.464754+0000","last_fullsized":"2024-04-22T01:12:17.464754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470357+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:12:17.470357+0000","last_peered":"2024-04-22T01:12:17.470357+0000","last_clean":"2024-04-22T01:12:17.470357+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:12:17.470357+0000","last_undegraded":"2024-04-22T01:12:17.470357+0000","last_fullsized":"2024-04-22T01:12:17.470357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467064+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:12:17.467064+0000","last_peered":"2024-04-22T01:12:17.467064+0000","last_clean":"2024-04-22T01:12:17.467064+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:12:17.467064+0000","last_undegraded":"2024-04-22T01:12:17.467064+0000","last_fullsized":"2024-04-22T01:12:17.467064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471987+0000","last_change":"2024-04-22T01:12:17.471987+0000","last_active":"2024-04-22T01:12:17.471987+0000","last_peered":"2024-04-22T01:12:17.471987+0000","last_clean":"2024-04-22T01:12:17.471987+0000","last_became_active":"2024-04-22T01:12:17.471369+0000","last_became_peered":"2024-04-22T01:12:17.471369+0000","last_unstale":"2024-04-22T01:12:17.471987+0000","last_undegraded":"2024-04-22T01:12:17.471987+0000","last_fullsized":"2024-04-22T01:12:17.471987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:20.856968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,8],"acting":[10,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467697+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:12:17.467697+0000","last_peered":"2024-04-22T01:12:17.467697+0000","last_clean":"2024-04-22T01:12:17.467697+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:12:17.467697+0000","last_undegraded":"2024-04-22T01:12:17.467697+0000","last_fullsized":"2024-04-22T01:12:17.467697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790341+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:12:17.790341+0000","last_peered":"2024-04-22T01:12:17.790341+0000","last_clean":"2024-04-22T01:12:17.790341+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:12:17.790341+0000","last_undegraded":"2024-04-22T01:12:17.790341+0000","last_fullsized":"2024-04-22T01:12:17.790341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464549+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:12:17.464549+0000","last_peered":"2024-04-22T01:12:17.464549+0000","last_clean":"2024-04-22T01:12:17.464549+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:12:17.464549+0000","last_undegraded":"2024-04-22T01:12:17.464549+0000","last_fullsized":"2024-04-22T01:12:17.464549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:38.991705+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:12:38.991705+0000","last_peered":"2024-04-22T01:12:38.991705+0000","last_clean":"2024-04-22T01:12:38.991705+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:12:38.991705+0000","last_undegraded":"2024-04-22T01:12:38.991705+0000","last_fullsized":"2024-04-22T01:12:38.991705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467870+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:12:17.467870+0000","last_peered":"2024-04-22T01:12:17.467870+0000","last_clean":"2024-04-22T01:12:17.467870+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:12:17.467870+0000","last_undegraded":"2024-04-22T01:12:17.467870+0000","last_fullsized":"2024-04-22T01:12:17.467870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464696+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:12:17.464696+0000","last_peered":"2024-04-22T01:12:17.464696+0000","last_clean":"2024-04-22T01:12:17.464696+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:12:17.464696+0000","last_undegraded":"2024-04-22T01:12:17.464696+0000","last_fullsized":"2024-04-22T01:12:17.464696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467236+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:12:17.467236+0000","last_peered":"2024-04-22T01:12:17.467236+0000","last_clean":"2024-04-22T01:12:17.467236+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:12:17.467236+0000","last_undegraded":"2024-04-22T01:12:17.467236+0000","last_fullsized":"2024-04-22T01:12:17.467236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468077+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:12:17.468077+0000","last_peered":"2024-04-22T01:12:17.468077+0000","last_clean":"2024-04-22T01:12:17.468077+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:12:17.468077+0000","last_undegraded":"2024-04-22T01:12:17.468077+0000","last_fullsized":"2024-04-22T01:12:17.468077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470610+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:12:17.470610+0000","last_peered":"2024-04-22T01:12:17.470610+0000","last_clean":"2024-04-22T01:12:17.470610+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:12:17.470610+0000","last_undegraded":"2024-04-22T01:12:17.470610+0000","last_fullsized":"2024-04-22T01:12:17.470610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:03.624385+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:13:03.624385+0000","last_peered":"2024-04-22T01:13:03.624385+0000","last_clean":"2024-04-22T01:13:03.624385+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:13:03.624385+0000","last_undegraded":"2024-04-22T01:13:03.624385+0000","last_fullsized":"2024-04-22T01:13:03.624385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2093,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468326+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:12:17.468326+0000","last_peered":"2024-04-22T01:12:17.468326+0000","last_clean":"2024-04-22T01:12:17.468326+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:12:17.468326+0000","last_undegraded":"2024-04-22T01:12:17.468326+0000","last_fullsized":"2024-04-22T01:12:17.468326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467471+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:12:17.467471+0000","last_peered":"2024-04-22T01:12:17.467471+0000","last_clean":"2024-04-22T01:12:17.467471+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:12:17.467471+0000","last_undegraded":"2024-04-22T01:12:17.467471+0000","last_fullsized":"2024-04-22T01:12:17.467471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463429+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:12:17.463429+0000","last_peered":"2024-04-22T01:12:17.463429+0000","last_clean":"2024-04-22T01:12:17.463429+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:12:17.463429+0000","last_undegraded":"2024-04-22T01:12:17.463429+0000","last_fullsized":"2024-04-22T01:12:17.463429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467776+0000","last_change":"2024-04-22T01:11:17.313900+0000","last_active":"2024-04-22T01:12:17.467776+0000","last_peered":"2024-04-22T01:12:17.467776+0000","last_clean":"2024-04-22T01:12:17.467776+0000","last_became_active":"2024-04-22T01:11:17.313493+0000","last_became_peered":"2024-04-22T01:11:17.313493+0000","last_unstale":"2024-04-22T01:12:17.467776+0000","last_undegraded":"2024-04-22T01:12:17.467776+0000","last_fullsized":"2024-04-22T01:12:17.467776+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:30:42.909373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462618+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:12:17.462618+0000","last_peered":"2024-04-22T01:12:17.462618+0000","last_clean":"2024-04-22T01:12:17.462618+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:12:17.462618+0000","last_undegraded":"2024-04-22T01:12:17.462618+0000","last_fullsized":"2024-04-22T01:12:17.462618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462388+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:12:17.462388+0000","last_peered":"2024-04-22T01:12:17.462388+0000","last_clean":"2024-04-22T01:12:17.462388+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:12:17.462388+0000","last_undegraded":"2024-04-22T01:12:17.462388+0000","last_fullsized":"2024-04-22T01:12:17.462388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468003+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:12:17.468003+0000","last_peered":"2024-04-22T01:12:17.468003+0000","last_clean":"2024-04-22T01:12:17.468003+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:12:17.468003+0000","last_undegraded":"2024-04-22T01:12:17.468003+0000","last_fullsized":"2024-04-22T01:12:17.468003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790421+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:12:17.790421+0000","last_peered":"2024-04-22T01:12:17.790421+0000","last_clean":"2024-04-22T01:12:17.790421+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:12:17.790421+0000","last_undegraded":"2024-04-22T01:12:17.790421+0000","last_fullsized":"2024-04-22T01:12:17.790421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470087+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:12:17.470087+0000","last_peered":"2024-04-22T01:12:17.470087+0000","last_clean":"2024-04-22T01:12:17.470087+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:12:17.470087+0000","last_undegraded":"2024-04-22T01:12:17.470087+0000","last_fullsized":"2024-04-22T01:12:17.470087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470224+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:12:17.470224+0000","last_peered":"2024-04-22T01:12:17.470224+0000","last_clean":"2024-04-22T01:12:17.470224+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:12:17.470224+0000","last_undegraded":"2024-04-22T01:12:17.470224+0000","last_fullsized":"2024-04-22T01:12:17.470224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467790+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:12:17.467790+0000","last_peered":"2024-04-22T01:12:17.467790+0000","last_clean":"2024-04-22T01:12:17.467790+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:12:17.467790+0000","last_undegraded":"2024-04-22T01:12:17.467790+0000","last_fullsized":"2024-04-22T01:12:17.467790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462189+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:12:17.462189+0000","last_peered":"2024-04-22T01:12:17.462189+0000","last_clean":"2024-04-22T01:12:17.462189+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:12:17.462189+0000","last_undegraded":"2024-04-22T01:12:17.462189+0000","last_fullsized":"2024-04-22T01:12:17.462189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790840+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:12:17.790840+0000","last_peered":"2024-04-22T01:12:17.790840+0000","last_clean":"2024-04-22T01:12:17.790840+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:12:17.790840+0000","last_undegraded":"2024-04-22T01:12:17.790840+0000","last_fullsized":"2024-04-22T01:12:17.790840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470692+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:12:17.470692+0000","last_peered":"2024-04-22T01:12:17.470692+0000","last_clean":"2024-04-22T01:12:17.470692+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:12:17.470692+0000","last_undegraded":"2024-04-22T01:12:17.470692+0000","last_fullsized":"2024-04-22T01:12:17.470692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468527+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:12:17.468527+0000","last_peered":"2024-04-22T01:12:17.468527+0000","last_clean":"2024-04-22T01:12:17.468527+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:12:17.468527+0000","last_undegraded":"2024-04-22T01:12:17.468527+0000","last_fullsized":"2024-04-22T01:12:17.468527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462784+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:12:17.462784+0000","last_peered":"2024-04-22T01:12:17.462784+0000","last_clean":"2024-04-22T01:12:17.462784+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:12:17.462784+0000","last_undegraded":"2024-04-22T01:12:17.462784+0000","last_fullsized":"2024-04-22T01:12:17.462784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467872+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:12:17.467872+0000","last_peered":"2024-04-22T01:12:17.467872+0000","last_clean":"2024-04-22T01:12:17.467872+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:12:17.467872+0000","last_undegraded":"2024-04-22T01:12:17.467872+0000","last_fullsized":"2024-04-22T01:12:17.467872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470799+0000","last_change":"2024-04-22T01:11:17.586313+0000","last_active":"2024-04-22T01:12:17.470799+0000","last_peered":"2024-04-22T01:12:17.470799+0000","last_clean":"2024-04-22T01:12:17.470799+0000","last_became_active":"2024-04-22T01:11:17.585705+0000","last_became_peered":"2024-04-22T01:11:17.585705+0000","last_unstale":"2024-04-22T01:12:17.470799+0000","last_undegraded":"2024-04-22T01:12:17.470799+0000","last_fullsized":"2024-04-22T01:12:17.470799+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:02.313774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462485+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:12:17.462485+0000","last_peered":"2024-04-22T01:12:17.462485+0000","last_clean":"2024-04-22T01:12:17.462485+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:12:17.462485+0000","last_undegraded":"2024-04-22T01:12:17.462485+0000","last_fullsized":"2024-04-22T01:12:17.462485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790617+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:12:17.790617+0000","last_peered":"2024-04-22T01:12:17.790617+0000","last_clean":"2024-04-22T01:12:17.790617+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:12:17.790617+0000","last_undegraded":"2024-04-22T01:12:17.790617+0000","last_fullsized":"2024-04-22T01:12:17.790617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69775,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1256,"num_read_kb":700,"num_write":257,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":672,"num_keys_recovered":7,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":217953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":521373,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1175552,"data_stored":1168157,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":524476,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":520086,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1117704,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495687,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":515884,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":583490,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":517872,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":584884,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":515592,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":519958,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1113369,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:13:07.328 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-22T01:13:07.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/426490667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc102230 msgr2=0x7f0cdc108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/426490667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc102230 0x7f0cdc108ac0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009ca0 tx=0x7f0cd402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/426490667 shutdown_connections 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/426490667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc102230 0x7f0cdc108ac0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/426490667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc101870 0x7f0cdc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/426490667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc100680 0x7f0cdc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/426490667 >> 172.21.15.38:0/426490667 conn(0x7f0cdc0fbbf0 msgr2=0x7f0cdc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:07.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/426490667 shutdown_connections 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/426490667 wait complete. 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 Processor -- start 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.467+0000 7f0ce3ca7640 1 -- start start 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 0x7f0cdc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc101870 0x7f0cdc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc102230 0x7f0cdc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc10c300 con 0x7f0cdc100680 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc10c180 con 0x7f0cdc102230 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc10c480 con 0x7f0cdc101870 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 0x7f0cdc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 0x7f0cdc197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37326/0 (socket says 172.21.15.38:37326) 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 -- 172.21.15.38:0/2950919408 learned_addr learned my addr 172.21.15.38:0/2950919408 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce2244640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc102230 0x7f0cdc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1242640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc101870 0x7f0cdc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 -- 172.21.15.38:0/2950919408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc101870 msgr2=0x7f0cdc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc101870 0x7f0cdc197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 -- 172.21.15.38:0/2950919408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc102230 msgr2=0x7f0cdc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc102230 0x7f0cdc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 -- 172.21.15.38:0/2950919408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc19f670 con 0x7f0cdc100680 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce2244640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc102230 0x7f0cdc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce1a43640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 0x7f0cdc197a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e9c0 tx=0x7f0ccc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc00cd80 con 0x7f0cdc100680 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cdc19f960 con 0x7f0cdc100680 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.468+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ccc0035f0 con 0x7f0cdc100680 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.469+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc010690 con 0x7f0cdc100680 2024-04-22T01:13:07.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.469+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cdc19feb0 con 0x7f0cdc100680 2024-04-22T01:13:07.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.470+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cdc1a00e0 con 0x7f0cdc100680 2024-04-22T01:13:07.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.471+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f0ccc0037a0 con 0x7f0cdc100680 2024-04-22T01:13:07.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.471+0000 7f0cd2ffd640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0cc80a2c20 0x7f0cc80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.472+0000 7f0ce1242640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0cc80a2c20 0x7f0cc80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.472+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f0ccc0cfa00 con 0x7f0cdc100680 2024-04-22T01:13:07.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.472+0000 7f0ce1242640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0cc80a2c20 0x7f0cc80a50e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc198f60 tx=0x7f0cd800a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:07.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.479+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0ccc09c4c0 con 0x7f0cdc100680 2024-04-22T01:13:07.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.670+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 --> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0cdc1a02b0 con 0x7f0cc80a2c20 2024-04-22T01:13:07.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.679+0000 7f0cd2ffd640 1 -- 172.21.15.38:0/2950919408 <== mgr.14152 v2:172.21.15.38:6800/3086698001 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340810 (secure 0 0 0) 0x7f0cdc1a02b0 con 0x7f0cc80a2c20 2024-04-22T01:13:07.682 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:07.684 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T01:13:07.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0cc80a2c20 msgr2=0x7f0cc80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0cc80a2c20 0x7f0cc80a50e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc198f60 tx=0x7f0cd800a250 comp rx=0 tx=0).stop 2024-04-22T01:13:07.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 msgr2=0x7f0cdc197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 0x7f0cdc197a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e9c0 tx=0x7f0ccc00ee80 comp rx=0 tx=0).stop 2024-04-22T01:13:07.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 shutdown_connections 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0cc80a2c20 0x7f0cc80a50e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cdc102230 0x7f0cdc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc101870 0x7f0cdc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.683+0000 7f0ce3ca7640 1 --2- 172.21.15.38:0/2950919408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cdc100680 0x7f0cdc197a50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.684+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 >> 172.21.15.38:0/2950919408 conn(0x7f0cdc0fbbf0 msgr2=0x7f0cdc0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.684+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 shutdown_connections 2024-04-22T01:13:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.684+0000 7f0ce3ca7640 1 -- 172.21.15.38:0/2950919408 wait complete. 2024-04-22T01:13:07.695 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-04-22T01:13:05.826500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660377,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1382,"num_read_kb":809,"num_write":440,"num_write_kb":3091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181408,"num_keys_recovered":7,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357636,"kb_used_data":12592,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584188,"statfs":{"total":1151940427776,"available":1151574208512,"internally_reserved":0,"allocated":12894208,"data_stored":8202845,"data_compressed":428511,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1398,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":52,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790906+0000","last_change":"2024-04-22T01:10:30.470238+0000","last_active":"2024-04-22T01:12:17.790906+0000","last_peered":"2024-04-22T01:12:17.790906+0000","last_clean":"2024-04-22T01:12:17.790906+0000","last_became_active":"2024-04-22T01:10:30.469601+0000","last_became_peered":"2024-04-22T01:10:30.469601+0000","last_unstale":"2024-04-22T01:12:17.790906+0000","last_undegraded":"2024-04-22T01:12:17.790906+0000","last_fullsized":"2024-04-22T01:12:17.790906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:04:16.593751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468337+0000","last_change":"2024-04-22T01:10:30.462694+0000","last_active":"2024-04-22T01:12:17.468337+0000","last_peered":"2024-04-22T01:12:17.468337+0000","last_clean":"2024-04-22T01:12:17.468337+0000","last_became_active":"2024-04-22T01:10:30.462400+0000","last_became_peered":"2024-04-22T01:10:30.462400+0000","last_unstale":"2024-04-22T01:12:17.468337+0000","last_undegraded":"2024-04-22T01:12:17.468337+0000","last_fullsized":"2024-04-22T01:12:17.468337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:23.973390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467910+0000","last_change":"2024-04-22T01:10:30.464445+0000","last_active":"2024-04-22T01:12:17.467910+0000","last_peered":"2024-04-22T01:12:17.467910+0000","last_clean":"2024-04-22T01:12:17.467910+0000","last_became_active":"2024-04-22T01:10:30.464127+0000","last_became_peered":"2024-04-22T01:10:30.464127+0000","last_unstale":"2024-04-22T01:12:17.467910+0000","last_undegraded":"2024-04-22T01:12:17.467910+0000","last_fullsized":"2024-04-22T01:12:17.467910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.398788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464881+0000","last_change":"2024-04-22T01:10:30.475396+0000","last_active":"2024-04-22T01:12:17.464881+0000","last_peered":"2024-04-22T01:12:17.464881+0000","last_clean":"2024-04-22T01:12:17.464881+0000","last_became_active":"2024-04-22T01:10:30.475102+0000","last_became_peered":"2024-04-22T01:10:30.475102+0000","last_unstale":"2024-04-22T01:12:17.464881+0000","last_undegraded":"2024-04-22T01:12:17.464881+0000","last_fullsized":"2024-04-22T01:12:17.464881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:58.377242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467393+0000","last_change":"2024-04-22T01:10:30.469651+0000","last_active":"2024-04-22T01:12:17.467393+0000","last_peered":"2024-04-22T01:12:17.467393+0000","last_clean":"2024-04-22T01:12:17.467393+0000","last_became_active":"2024-04-22T01:10:30.469352+0000","last_became_peered":"2024-04-22T01:10:30.469352+0000","last_unstale":"2024-04-22T01:12:17.467393+0000","last_undegraded":"2024-04-22T01:12:17.467393+0000","last_fullsized":"2024-04-22T01:12:17.467393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:14.851469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466033+0000","last_change":"2024-04-22T01:10:30.469660+0000","last_active":"2024-04-22T01:12:17.466033+0000","last_peered":"2024-04-22T01:12:17.466033+0000","last_clean":"2024-04-22T01:12:17.466033+0000","last_became_active":"2024-04-22T01:10:30.469313+0000","last_became_peered":"2024-04-22T01:10:30.469313+0000","last_unstale":"2024-04-22T01:12:17.466033+0000","last_undegraded":"2024-04-22T01:12:17.466033+0000","last_fullsized":"2024-04-22T01:12:17.466033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:34:26.000089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467329+0000","last_change":"2024-04-22T01:10:30.472208+0000","last_active":"2024-04-22T01:12:17.467329+0000","last_peered":"2024-04-22T01:12:17.467329+0000","last_clean":"2024-04-22T01:12:17.467329+0000","last_became_active":"2024-04-22T01:10:30.471854+0000","last_became_peered":"2024-04-22T01:10:30.471854+0000","last_unstale":"2024-04-22T01:12:17.467329+0000","last_undegraded":"2024-04-22T01:12:17.467329+0000","last_fullsized":"2024-04-22T01:12:17.467329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:52.727216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467174+0000","last_change":"2024-04-22T01:10:30.473972+0000","last_active":"2024-04-22T01:12:17.467174+0000","last_peered":"2024-04-22T01:12:17.467174+0000","last_clean":"2024-04-22T01:12:17.467174+0000","last_became_active":"2024-04-22T01:10:30.473021+0000","last_became_peered":"2024-04-22T01:10:30.473021+0000","last_unstale":"2024-04-22T01:12:17.467174+0000","last_undegraded":"2024-04-22T01:12:17.467174+0000","last_fullsized":"2024-04-22T01:12:17.467174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:11:30.568578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471434+0000","last_change":"2024-04-22T01:10:30.476472+0000","last_active":"2024-04-22T01:12:17.471434+0000","last_peered":"2024-04-22T01:12:17.471434+0000","last_clean":"2024-04-22T01:12:17.471434+0000","last_became_active":"2024-04-22T01:10:30.476155+0000","last_became_peered":"2024-04-22T01:10:30.476155+0000","last_unstale":"2024-04-22T01:12:17.471434+0000","last_undegraded":"2024-04-22T01:12:17.471434+0000","last_fullsized":"2024-04-22T01:12:17.471434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:36:32.317702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466554+0000","last_change":"2024-04-22T01:10:30.459178+0000","last_active":"2024-04-22T01:12:17.466554+0000","last_peered":"2024-04-22T01:12:17.466554+0000","last_clean":"2024-04-22T01:12:17.466554+0000","last_became_active":"2024-04-22T01:10:30.458641+0000","last_became_peered":"2024-04-22T01:10:30.458641+0000","last_unstale":"2024-04-22T01:12:17.466554+0000","last_undegraded":"2024-04-22T01:12:17.466554+0000","last_fullsized":"2024-04-22T01:12:17.466554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:42:11.478263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468253+0000","last_change":"2024-04-22T01:10:30.477530+0000","last_active":"2024-04-22T01:12:17.468253+0000","last_peered":"2024-04-22T01:12:17.468253+0000","last_clean":"2024-04-22T01:12:17.468253+0000","last_became_active":"2024-04-22T01:10:30.477227+0000","last_became_peered":"2024-04-22T01:10:30.477227+0000","last_unstale":"2024-04-22T01:12:17.468253+0000","last_undegraded":"2024-04-22T01:12:17.468253+0000","last_fullsized":"2024-04-22T01:12:17.468253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:31.512973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464762+0000","last_change":"2024-04-22T01:10:30.472537+0000","last_active":"2024-04-22T01:12:17.464762+0000","last_peered":"2024-04-22T01:12:17.464762+0000","last_clean":"2024-04-22T01:12:17.464762+0000","last_became_active":"2024-04-22T01:10:30.472144+0000","last_became_peered":"2024-04-22T01:10:30.472144+0000","last_unstale":"2024-04-22T01:12:17.464762+0000","last_undegraded":"2024-04-22T01:12:17.464762+0000","last_fullsized":"2024-04-22T01:12:17.464762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:26.677741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462419+0000","last_change":"2024-04-22T01:11:17.312524+0000","last_active":"2024-04-22T01:12:17.462419+0000","last_peered":"2024-04-22T01:12:17.462419+0000","last_clean":"2024-04-22T01:12:17.462419+0000","last_became_active":"2024-04-22T01:11:17.312068+0000","last_became_peered":"2024-04-22T01:11:17.312068+0000","last_unstale":"2024-04-22T01:12:17.462419+0000","last_undegraded":"2024-04-22T01:12:17.462419+0000","last_fullsized":"2024-04-22T01:12:17.462419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:47:17.243712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467343+0000","last_change":"2024-04-22T01:10:30.469103+0000","last_active":"2024-04-22T01:12:17.467343+0000","last_peered":"2024-04-22T01:12:17.467343+0000","last_clean":"2024-04-22T01:12:17.467343+0000","last_became_active":"2024-04-22T01:10:30.468164+0000","last_became_peered":"2024-04-22T01:10:30.468164+0000","last_unstale":"2024-04-22T01:12:17.467343+0000","last_undegraded":"2024-04-22T01:12:17.467343+0000","last_fullsized":"2024-04-22T01:12:17.467343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:24.325011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467849+0000","last_change":"2024-04-22T01:10:30.469557+0000","last_active":"2024-04-22T01:12:17.467849+0000","last_peered":"2024-04-22T01:12:17.467849+0000","last_clean":"2024-04-22T01:12:17.467849+0000","last_became_active":"2024-04-22T01:10:30.469230+0000","last_became_peered":"2024-04-22T01:10:30.469230+0000","last_unstale":"2024-04-22T01:12:17.467849+0000","last_undegraded":"2024-04-22T01:12:17.467849+0000","last_fullsized":"2024-04-22T01:12:17.467849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:10:13.010238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470498+0000","last_change":"2024-04-22T01:10:30.466216+0000","last_active":"2024-04-22T01:12:17.470498+0000","last_peered":"2024-04-22T01:12:17.470498+0000","last_clean":"2024-04-22T01:12:17.470498+0000","last_became_active":"2024-04-22T01:10:30.465929+0000","last_became_peered":"2024-04-22T01:10:30.465929+0000","last_unstale":"2024-04-22T01:12:17.470498+0000","last_undegraded":"2024-04-22T01:12:17.470498+0000","last_fullsized":"2024-04-22T01:12:17.470498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:42:21.981025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790752+0000","last_change":"2024-04-22T01:10:30.470511+0000","last_active":"2024-04-22T01:12:17.790752+0000","last_peered":"2024-04-22T01:12:17.790752+0000","last_clean":"2024-04-22T01:12:17.790752+0000","last_became_active":"2024-04-22T01:10:30.469957+0000","last_became_peered":"2024-04-22T01:10:30.469957+0000","last_unstale":"2024-04-22T01:12:17.790752+0000","last_undegraded":"2024-04-22T01:12:17.790752+0000","last_fullsized":"2024-04-22T01:12:17.790752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:56:28.879731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465060+0000","last_change":"2024-04-22T01:10:30.465939+0000","last_active":"2024-04-22T01:12:17.465060+0000","last_peered":"2024-04-22T01:12:17.465060+0000","last_clean":"2024-04-22T01:12:17.465060+0000","last_became_active":"2024-04-22T01:10:30.465612+0000","last_became_peered":"2024-04-22T01:10:30.465612+0000","last_unstale":"2024-04-22T01:12:17.465060+0000","last_undegraded":"2024-04-22T01:12:17.465060+0000","last_fullsized":"2024-04-22T01:12:17.465060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:49.319678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470457+0000","last_change":"2024-04-22T01:10:30.466412+0000","last_active":"2024-04-22T01:12:17.470457+0000","last_peered":"2024-04-22T01:12:17.470457+0000","last_clean":"2024-04-22T01:12:17.470457+0000","last_became_active":"2024-04-22T01:10:30.466097+0000","last_became_peered":"2024-04-22T01:10:30.466097+0000","last_unstale":"2024-04-22T01:12:17.470457+0000","last_undegraded":"2024-04-22T01:12:17.470457+0000","last_fullsized":"2024-04-22T01:12:17.470457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:03:13.648342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470430+0000","last_change":"2024-04-22T01:11:17.586121+0000","last_active":"2024-04-22T01:12:17.470430+0000","last_peered":"2024-04-22T01:12:17.470430+0000","last_clean":"2024-04-22T01:12:17.470430+0000","last_became_active":"2024-04-22T01:11:17.585619+0000","last_became_peered":"2024-04-22T01:11:17.585619+0000","last_unstale":"2024-04-22T01:12:17.470430+0000","last_undegraded":"2024-04-22T01:12:17.470430+0000","last_fullsized":"2024-04-22T01:12:17.470430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:12.987810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:59.529087+0000","last_change":"2024-04-22T01:12:17.484778+0000","last_active":"2024-04-22T01:12:59.529087+0000","last_peered":"2024-04-22T01:12:59.529087+0000","last_clean":"2024-04-22T01:12:59.529087+0000","last_became_active":"2024-04-22T01:12:17.472811+0000","last_became_peered":"2024-04-22T01:12:17.472811+0000","last_unstale":"2024-04-22T01:12:59.529087+0000","last_undegraded":"2024-04-22T01:12:59.529087+0000","last_fullsized":"2024-04-22T01:12:59.529087+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:07.118642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467391+0000","last_change":"2024-04-22T01:10:30.474041+0000","last_active":"2024-04-22T01:12:17.467391+0000","last_peered":"2024-04-22T01:12:17.467391+0000","last_clean":"2024-04-22T01:12:17.467391+0000","last_became_active":"2024-04-22T01:10:30.471943+0000","last_became_peered":"2024-04-22T01:10:30.471943+0000","last_unstale":"2024-04-22T01:12:17.467391+0000","last_undegraded":"2024-04-22T01:12:17.467391+0000","last_fullsized":"2024-04-22T01:12:17.467391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:41:36.229699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464207+0000","last_change":"2024-04-22T01:10:34.671241+0000","last_active":"2024-04-22T01:12:17.464207+0000","last_peered":"2024-04-22T01:12:17.464207+0000","last_clean":"2024-04-22T01:12:17.464207+0000","last_became_active":"2024-04-22T01:10:34.670708+0000","last_became_peered":"2024-04-22T01:10:34.670708+0000","last_unstale":"2024-04-22T01:12:17.464207+0000","last_undegraded":"2024-04-22T01:12:17.464207+0000","last_fullsized":"2024-04-22T01:12:17.464207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:42:01.417893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468644+0000","last_change":"2024-04-22T01:10:29.595313+0000","last_active":"2024-04-22T01:12:17.468644+0000","last_peered":"2024-04-22T01:12:17.468644+0000","last_clean":"2024-04-22T01:12:17.468644+0000","last_became_active":"2024-04-22T01:10:29.595013+0000","last_became_peered":"2024-04-22T01:10:29.595013+0000","last_unstale":"2024-04-22T01:12:17.468644+0000","last_undegraded":"2024-04-22T01:12:17.468644+0000","last_fullsized":"2024-04-22T01:12:17.468644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:27:37.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468490+0000","last_change":"2024-04-22T01:10:30.468872+0000","last_active":"2024-04-22T01:12:17.468490+0000","last_peered":"2024-04-22T01:12:17.468490+0000","last_clean":"2024-04-22T01:12:17.468490+0000","last_became_active":"2024-04-22T01:10:30.468604+0000","last_became_peered":"2024-04-22T01:10:30.468604+0000","last_unstale":"2024-04-22T01:12:17.468490+0000","last_undegraded":"2024-04-22T01:12:17.468490+0000","last_fullsized":"2024-04-22T01:12:17.468490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:10:09.203100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464299+0000","last_change":"2024-04-22T01:10:34.671970+0000","last_active":"2024-04-22T01:12:17.464299+0000","last_peered":"2024-04-22T01:12:17.464299+0000","last_clean":"2024-04-22T01:12:17.464299+0000","last_became_active":"2024-04-22T01:10:34.670785+0000","last_became_peered":"2024-04-22T01:10:34.670785+0000","last_unstale":"2024-04-22T01:12:17.464299+0000","last_undegraded":"2024-04-22T01:12:17.464299+0000","last_fullsized":"2024-04-22T01:12:17.464299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:57:31.520805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467663+0000","last_change":"2024-04-22T01:10:29.595498+0000","last_active":"2024-04-22T01:12:17.467663+0000","last_peered":"2024-04-22T01:12:17.467663+0000","last_clean":"2024-04-22T01:12:17.467663+0000","last_became_active":"2024-04-22T01:10:29.594948+0000","last_became_peered":"2024-04-22T01:10:29.594948+0000","last_unstale":"2024-04-22T01:12:17.467663+0000","last_undegraded":"2024-04-22T01:12:17.467663+0000","last_fullsized":"2024-04-22T01:12:17.467663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:41:04.827714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470754+0000","last_change":"2024-04-22T01:10:30.467588+0000","last_active":"2024-04-22T01:12:17.470754+0000","last_peered":"2024-04-22T01:12:17.470754+0000","last_clean":"2024-04-22T01:12:17.470754+0000","last_became_active":"2024-04-22T01:10:30.467280+0000","last_became_peered":"2024-04-22T01:10:30.467280+0000","last_unstale":"2024-04-22T01:12:17.470754+0000","last_undegraded":"2024-04-22T01:12:17.470754+0000","last_fullsized":"2024-04-22T01:12:17.470754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:28:43.688221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464253+0000","last_change":"2024-04-22T01:10:34.673599+0000","last_active":"2024-04-22T01:12:17.464253+0000","last_peered":"2024-04-22T01:12:17.464253+0000","last_clean":"2024-04-22T01:12:17.464253+0000","last_became_active":"2024-04-22T01:10:34.672979+0000","last_became_peered":"2024-04-22T01:10:34.672979+0000","last_unstale":"2024-04-22T01:12:17.464253+0000","last_undegraded":"2024-04-22T01:12:17.464253+0000","last_fullsized":"2024-04-22T01:12:17.464253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:15.099616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467272+0000","last_change":"2024-04-22T01:10:29.588638+0000","last_active":"2024-04-22T01:12:17.467272+0000","last_peered":"2024-04-22T01:12:17.467272+0000","last_clean":"2024-04-22T01:12:17.467272+0000","last_became_active":"2024-04-22T01:10:29.588239+0000","last_became_peered":"2024-04-22T01:10:29.588239+0000","last_unstale":"2024-04-22T01:12:17.467272+0000","last_undegraded":"2024-04-22T01:12:17.467272+0000","last_fullsized":"2024-04-22T01:12:17.467272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:45.972575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463194+0000","last_change":"2024-04-22T01:10:30.468007+0000","last_active":"2024-04-22T01:12:17.463194+0000","last_peered":"2024-04-22T01:12:17.463194+0000","last_clean":"2024-04-22T01:12:17.463194+0000","last_became_active":"2024-04-22T01:10:30.467680+0000","last_became_peered":"2024-04-22T01:10:30.467680+0000","last_unstale":"2024-04-22T01:12:17.463194+0000","last_undegraded":"2024-04-22T01:12:17.463194+0000","last_fullsized":"2024-04-22T01:12:17.463194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:22:10.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470258+0000","last_change":"2024-04-22T01:10:34.666541+0000","last_active":"2024-04-22T01:12:17.470258+0000","last_peered":"2024-04-22T01:12:17.470258+0000","last_clean":"2024-04-22T01:12:17.470258+0000","last_became_active":"2024-04-22T01:10:34.665685+0000","last_became_peered":"2024-04-22T01:10:34.665685+0000","last_unstale":"2024-04-22T01:12:17.470258+0000","last_undegraded":"2024-04-22T01:12:17.470258+0000","last_fullsized":"2024-04-22T01:12:17.470258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:38.658971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:03.625138+0000","last_change":"2024-04-22T01:10:29.593260+0000","last_active":"2024-04-22T01:13:03.625138+0000","last_peered":"2024-04-22T01:13:03.625138+0000","last_clean":"2024-04-22T01:13:03.625138+0000","last_became_active":"2024-04-22T01:10:29.592902+0000","last_became_peered":"2024-04-22T01:10:29.592902+0000","last_unstale":"2024-04-22T01:13:03.625138+0000","last_undegraded":"2024-04-22T01:13:03.625138+0000","last_fullsized":"2024-04-22T01:13:03.625138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:27.678644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466990+0000","last_change":"2024-04-22T01:10:30.465048+0000","last_active":"2024-04-22T01:12:17.466990+0000","last_peered":"2024-04-22T01:12:17.466990+0000","last_clean":"2024-04-22T01:12:17.466990+0000","last_became_active":"2024-04-22T01:10:30.464720+0000","last_became_peered":"2024-04-22T01:10:30.464720+0000","last_unstale":"2024-04-22T01:12:17.466990+0000","last_undegraded":"2024-04-22T01:12:17.466990+0000","last_fullsized":"2024-04-22T01:12:17.466990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:49:55.719024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470301+0000","last_change":"2024-04-22T01:10:34.666055+0000","last_active":"2024-04-22T01:12:17.470301+0000","last_peered":"2024-04-22T01:12:17.470301+0000","last_clean":"2024-04-22T01:12:17.470301+0000","last_became_active":"2024-04-22T01:10:34.665608+0000","last_became_peered":"2024-04-22T01:10:34.665608+0000","last_unstale":"2024-04-22T01:12:17.470301+0000","last_undegraded":"2024-04-22T01:12:17.470301+0000","last_fullsized":"2024-04-22T01:12:17.470301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:34.197510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465883+0000","last_change":"2024-04-22T01:10:29.592264+0000","last_active":"2024-04-22T01:12:17.465883+0000","last_peered":"2024-04-22T01:12:17.465883+0000","last_clean":"2024-04-22T01:12:17.465883+0000","last_became_active":"2024-04-22T01:10:29.591854+0000","last_became_peered":"2024-04-22T01:10:29.591854+0000","last_unstale":"2024-04-22T01:12:17.465883+0000","last_undegraded":"2024-04-22T01:12:17.465883+0000","last_fullsized":"2024-04-22T01:12:17.465883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:00:56.531416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463275+0000","last_change":"2024-04-22T01:10:30.466623+0000","last_active":"2024-04-22T01:12:17.463275+0000","last_peered":"2024-04-22T01:12:17.463275+0000","last_clean":"2024-04-22T01:12:17.463275+0000","last_became_active":"2024-04-22T01:10:30.466302+0000","last_became_peered":"2024-04-22T01:10:30.466302+0000","last_unstale":"2024-04-22T01:12:17.463275+0000","last_undegraded":"2024-04-22T01:12:17.463275+0000","last_fullsized":"2024-04-22T01:12:17.463275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:11.024416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790698+0000","last_change":"2024-04-22T01:10:34.659131+0000","last_active":"2024-04-22T01:12:17.790698+0000","last_peered":"2024-04-22T01:12:17.790698+0000","last_clean":"2024-04-22T01:12:17.790698+0000","last_became_active":"2024-04-22T01:10:34.658565+0000","last_became_peered":"2024-04-22T01:10:34.658565+0000","last_unstale":"2024-04-22T01:12:17.790698+0000","last_undegraded":"2024-04-22T01:12:17.790698+0000","last_fullsized":"2024-04-22T01:12:17.790698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:36:48.904420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.461932+0000","last_change":"2024-04-22T01:10:29.583612+0000","last_active":"2024-04-22T01:12:17.461932+0000","last_peered":"2024-04-22T01:12:17.461932+0000","last_clean":"2024-04-22T01:12:17.461932+0000","last_became_active":"2024-04-22T01:10:29.583260+0000","last_became_peered":"2024-04-22T01:10:29.583260+0000","last_unstale":"2024-04-22T01:12:17.461932+0000","last_undegraded":"2024-04-22T01:12:17.461932+0000","last_fullsized":"2024-04-22T01:12:17.461932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:49.401257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467441+0000","last_change":"2024-04-22T01:10:30.469950+0000","last_active":"2024-04-22T01:12:17.467441+0000","last_peered":"2024-04-22T01:12:17.467441+0000","last_clean":"2024-04-22T01:12:17.467441+0000","last_became_active":"2024-04-22T01:10:30.466998+0000","last_became_peered":"2024-04-22T01:10:30.466998+0000","last_unstale":"2024-04-22T01:12:17.467441+0000","last_undegraded":"2024-04-22T01:12:17.467441+0000","last_fullsized":"2024-04-22T01:12:17.467441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:14.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468371+0000","last_change":"2024-04-22T01:10:34.662738+0000","last_active":"2024-04-22T01:12:17.468371+0000","last_peered":"2024-04-22T01:12:17.468371+0000","last_clean":"2024-04-22T01:12:17.468371+0000","last_became_active":"2024-04-22T01:10:34.662296+0000","last_became_peered":"2024-04-22T01:10:34.662296+0000","last_unstale":"2024-04-22T01:12:17.468371+0000","last_undegraded":"2024-04-22T01:12:17.468371+0000","last_fullsized":"2024-04-22T01:12:17.468371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:46:14.298980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462747+0000","last_change":"2024-04-22T01:10:29.585867+0000","last_active":"2024-04-22T01:12:17.462747+0000","last_peered":"2024-04-22T01:12:17.462747+0000","last_clean":"2024-04-22T01:12:17.462747+0000","last_became_active":"2024-04-22T01:10:29.585566+0000","last_became_peered":"2024-04-22T01:10:29.585566+0000","last_unstale":"2024-04-22T01:12:17.462747+0000","last_undegraded":"2024-04-22T01:12:17.462747+0000","last_fullsized":"2024-04-22T01:12:17.462747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:24.326326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464591+0000","last_change":"2024-04-22T01:10:30.467091+0000","last_active":"2024-04-22T01:12:17.464591+0000","last_peered":"2024-04-22T01:12:17.464591+0000","last_clean":"2024-04-22T01:12:17.464591+0000","last_became_active":"2024-04-22T01:10:30.466778+0000","last_became_peered":"2024-04-22T01:10:30.466778+0000","last_unstale":"2024-04-22T01:12:17.464591+0000","last_undegraded":"2024-04-22T01:12:17.464591+0000","last_fullsized":"2024-04-22T01:12:17.464591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:43.851878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467587+0000","last_change":"2024-04-22T01:10:34.675905+0000","last_active":"2024-04-22T01:12:17.467587+0000","last_peered":"2024-04-22T01:12:17.467587+0000","last_clean":"2024-04-22T01:12:17.467587+0000","last_became_active":"2024-04-22T01:10:34.675497+0000","last_became_peered":"2024-04-22T01:10:34.675497+0000","last_unstale":"2024-04-22T01:12:17.467587+0000","last_undegraded":"2024-04-22T01:12:17.467587+0000","last_fullsized":"2024-04-22T01:12:17.467587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:52:55.972160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":279,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.619054+0000","last_change":"2024-04-22T01:10:29.585915+0000","last_active":"2024-04-22T01:13:00.619054+0000","last_peered":"2024-04-22T01:13:00.619054+0000","last_clean":"2024-04-22T01:13:00.619054+0000","last_became_active":"2024-04-22T01:10:29.585557+0000","last_became_peered":"2024-04-22T01:10:29.585557+0000","last_unstale":"2024-04-22T01:13:00.619054+0000","last_undegraded":"2024-04-22T01:13:00.619054+0000","last_fullsized":"2024-04-22T01:13:00.619054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:53.128260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471276+0000","last_change":"2024-04-22T01:10:30.469867+0000","last_active":"2024-04-22T01:12:17.471276+0000","last_peered":"2024-04-22T01:12:17.471276+0000","last_clean":"2024-04-22T01:12:17.471276+0000","last_became_active":"2024-04-22T01:10:30.469080+0000","last_became_peered":"2024-04-22T01:10:30.469080+0000","last_unstale":"2024-04-22T01:12:17.471276+0000","last_undegraded":"2024-04-22T01:12:17.471276+0000","last_fullsized":"2024-04-22T01:12:17.471276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:34:48.893070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464653+0000","last_change":"2024-04-22T01:10:34.671786+0000","last_active":"2024-04-22T01:12:17.464653+0000","last_peered":"2024-04-22T01:12:17.464653+0000","last_clean":"2024-04-22T01:12:17.464653+0000","last_became_active":"2024-04-22T01:10:34.671302+0000","last_became_peered":"2024-04-22T01:10:34.671302+0000","last_unstale":"2024-04-22T01:12:17.464653+0000","last_undegraded":"2024-04-22T01:12:17.464653+0000","last_fullsized":"2024-04-22T01:12:17.464653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:18.739312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.477748+0000","last_change":"2024-04-22T01:12:17.477108+0000","last_active":"2024-04-22T01:12:17.477748+0000","last_peered":"2024-04-22T01:12:17.477748+0000","last_clean":"2024-04-22T01:12:17.477748+0000","last_became_active":"2024-04-22T01:12:17.471865+0000","last_became_peered":"2024-04-22T01:12:17.471865+0000","last_unstale":"2024-04-22T01:12:17.477748+0000","last_undegraded":"2024-04-22T01:12:17.477748+0000","last_fullsized":"2024-04-22T01:12:17.477748+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:49:55.734197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468386+0000","last_change":"2024-04-22T01:10:30.472027+0000","last_active":"2024-04-22T01:12:17.468386+0000","last_peered":"2024-04-22T01:12:17.468386+0000","last_clean":"2024-04-22T01:12:17.468386+0000","last_became_active":"2024-04-22T01:10:30.471724+0000","last_became_peered":"2024-04-22T01:10:30.471724+0000","last_unstale":"2024-04-22T01:12:17.468386+0000","last_undegraded":"2024-04-22T01:12:17.468386+0000","last_fullsized":"2024-04-22T01:12:17.468386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:34.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464504+0000","last_change":"2024-04-22T01:11:17.586221+0000","last_active":"2024-04-22T01:12:17.464504+0000","last_peered":"2024-04-22T01:12:17.464504+0000","last_clean":"2024-04-22T01:12:17.464504+0000","last_became_active":"2024-04-22T01:11:17.585689+0000","last_became_peered":"2024-04-22T01:11:17.585689+0000","last_unstale":"2024-04-22T01:12:17.464504+0000","last_undegraded":"2024-04-22T01:12:17.464504+0000","last_fullsized":"2024-04-22T01:12:17.464504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:40.327272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463285+0000","last_change":"2024-04-22T01:10:29.591305+0000","last_active":"2024-04-22T01:12:17.463285+0000","last_peered":"2024-04-22T01:12:17.463285+0000","last_clean":"2024-04-22T01:12:17.463285+0000","last_became_active":"2024-04-22T01:10:29.590990+0000","last_became_peered":"2024-04-22T01:10:29.590990+0000","last_unstale":"2024-04-22T01:12:17.463285+0000","last_undegraded":"2024-04-22T01:12:17.463285+0000","last_fullsized":"2024-04-22T01:12:17.463285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:57.391800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470885+0000","last_change":"2024-04-22T01:11:17.585668+0000","last_active":"2024-04-22T01:12:17.470885+0000","last_peered":"2024-04-22T01:12:17.470885+0000","last_clean":"2024-04-22T01:12:17.470885+0000","last_became_active":"2024-04-22T01:11:17.585217+0000","last_became_peered":"2024-04-22T01:11:17.585217+0000","last_unstale":"2024-04-22T01:12:17.470885+0000","last_undegraded":"2024-04-22T01:12:17.470885+0000","last_fullsized":"2024-04-22T01:12:17.470885+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:24.007210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464709+0000","last_change":"2024-04-22T01:10:34.661452+0000","last_active":"2024-04-22T01:12:17.464709+0000","last_peered":"2024-04-22T01:12:17.464709+0000","last_clean":"2024-04-22T01:12:17.464709+0000","last_became_active":"2024-04-22T01:10:34.661004+0000","last_became_peered":"2024-04-22T01:10:34.661004+0000","last_unstale":"2024-04-22T01:12:17.464709+0000","last_undegraded":"2024-04-22T01:12:17.464709+0000","last_fullsized":"2024-04-22T01:12:17.464709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:17.229050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'38","reported_seq":318,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:01.485731+0000","last_change":"2024-04-22T01:10:29.591361+0000","last_active":"2024-04-22T01:13:01.485731+0000","last_peered":"2024-04-22T01:13:01.485731+0000","last_clean":"2024-04-22T01:13:01.485731+0000","last_became_active":"2024-04-22T01:10:29.591000+0000","last_became_peered":"2024-04-22T01:10:29.591000+0000","last_unstale":"2024-04-22T01:13:01.485731+0000","last_undegraded":"2024-04-22T01:13:01.485731+0000","last_fullsized":"2024-04-22T01:13:01.485731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:39:58.171105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61199,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":249,"num_read_kb":184,"num_write":40,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467503+0000","last_change":"2024-04-22T01:10:30.472960+0000","last_active":"2024-04-22T01:12:17.467503+0000","last_peered":"2024-04-22T01:12:17.467503+0000","last_clean":"2024-04-22T01:12:17.467503+0000","last_became_active":"2024-04-22T01:10:30.472318+0000","last_became_peered":"2024-04-22T01:10:30.472318+0000","last_unstale":"2024-04-22T01:12:17.467503+0000","last_undegraded":"2024-04-22T01:12:17.467503+0000","last_fullsized":"2024-04-22T01:12:17.467503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:57.263326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463529+0000","last_change":"2024-04-22T01:10:34.672444+0000","last_active":"2024-04-22T01:12:17.463529+0000","last_peered":"2024-04-22T01:12:17.463529+0000","last_clean":"2024-04-22T01:12:17.463529+0000","last_became_active":"2024-04-22T01:10:34.671849+0000","last_became_peered":"2024-04-22T01:10:34.671849+0000","last_unstale":"2024-04-22T01:12:17.463529+0000","last_undegraded":"2024-04-22T01:12:17.463529+0000","last_fullsized":"2024-04-22T01:12:17.463529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:10:05.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470394+0000","last_change":"2024-04-22T01:10:29.587919+0000","last_active":"2024-04-22T01:12:17.470394+0000","last_peered":"2024-04-22T01:12:17.470394+0000","last_clean":"2024-04-22T01:12:17.470394+0000","last_became_active":"2024-04-22T01:10:29.587617+0000","last_became_peered":"2024-04-22T01:10:29.587617+0000","last_unstale":"2024-04-22T01:12:17.470394+0000","last_undegraded":"2024-04-22T01:12:17.470394+0000","last_fullsized":"2024-04-22T01:12:17.470394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:20:12.075286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462066+0000","last_change":"2024-04-22T01:10:30.467340+0000","last_active":"2024-04-22T01:12:17.462066+0000","last_peered":"2024-04-22T01:12:17.462066+0000","last_clean":"2024-04-22T01:12:17.462066+0000","last_became_active":"2024-04-22T01:10:30.466991+0000","last_became_peered":"2024-04-22T01:10:30.466991+0000","last_unstale":"2024-04-22T01:12:17.462066+0000","last_undegraded":"2024-04-22T01:12:17.462066+0000","last_fullsized":"2024-04-22T01:12:17.462066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:40:04.710423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790993+0000","last_change":"2024-04-22T01:10:34.663094+0000","last_active":"2024-04-22T01:12:17.790993+0000","last_peered":"2024-04-22T01:12:17.790993+0000","last_clean":"2024-04-22T01:12:17.790993+0000","last_became_active":"2024-04-22T01:10:34.662693+0000","last_became_peered":"2024-04-22T01:10:34.662693+0000","last_unstale":"2024-04-22T01:12:17.790993+0000","last_undegraded":"2024-04-22T01:12:17.790993+0000","last_fullsized":"2024-04-22T01:12:17.790993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:34:18.362791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463155+0000","last_change":"2024-04-22T01:10:29.592851+0000","last_active":"2024-04-22T01:12:17.463155+0000","last_peered":"2024-04-22T01:12:17.463155+0000","last_clean":"2024-04-22T01:12:17.463155+0000","last_became_active":"2024-04-22T01:10:29.591666+0000","last_became_peered":"2024-04-22T01:10:29.591666+0000","last_unstale":"2024-04-22T01:12:17.463155+0000","last_undegraded":"2024-04-22T01:12:17.463155+0000","last_fullsized":"2024-04-22T01:12:17.463155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:44:18.370074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467829+0000","last_change":"2024-04-22T01:10:30.473314+0000","last_active":"2024-04-22T01:12:17.467829+0000","last_peered":"2024-04-22T01:12:17.467829+0000","last_clean":"2024-04-22T01:12:17.467829+0000","last_became_active":"2024-04-22T01:10:30.472875+0000","last_became_peered":"2024-04-22T01:10:30.472875+0000","last_unstale":"2024-04-22T01:12:17.467829+0000","last_undegraded":"2024-04-22T01:12:17.467829+0000","last_fullsized":"2024-04-22T01:12:17.467829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:47.664251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464953+0000","last_change":"2024-04-22T01:10:34.671948+0000","last_active":"2024-04-22T01:12:17.464953+0000","last_peered":"2024-04-22T01:12:17.464953+0000","last_clean":"2024-04-22T01:12:17.464953+0000","last_became_active":"2024-04-22T01:10:34.671530+0000","last_became_peered":"2024-04-22T01:10:34.671530+0000","last_unstale":"2024-04-22T01:12:17.464953+0000","last_undegraded":"2024-04-22T01:12:17.464953+0000","last_fullsized":"2024-04-22T01:12:17.464953+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:19.838206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.654949+0000","last_change":"2024-04-22T01:10:29.586621+0000","last_active":"2024-04-22T01:13:00.654949+0000","last_peered":"2024-04-22T01:13:00.654949+0000","last_clean":"2024-04-22T01:13:00.654949+0000","last_became_active":"2024-04-22T01:10:29.586303+0000","last_became_peered":"2024-04-22T01:10:29.586303+0000","last_unstale":"2024-04-22T01:13:00.654949+0000","last_undegraded":"2024-04-22T01:13:00.654949+0000","last_fullsized":"2024-04-22T01:13:00.654949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:26:08.895285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467681+0000","last_change":"2024-04-22T01:11:17.314098+0000","last_active":"2024-04-22T01:12:17.467681+0000","last_peered":"2024-04-22T01:12:17.467681+0000","last_clean":"2024-04-22T01:12:17.467681+0000","last_became_active":"2024-04-22T01:11:17.313663+0000","last_became_peered":"2024-04-22T01:11:17.313663+0000","last_unstale":"2024-04-22T01:12:17.467681+0000","last_undegraded":"2024-04-22T01:12:17.467681+0000","last_fullsized":"2024-04-22T01:12:17.467681+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:20:03.826857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462323+0000","last_change":"2024-04-22T01:10:34.665985+0000","last_active":"2024-04-22T01:12:17.462323+0000","last_peered":"2024-04-22T01:12:17.462323+0000","last_clean":"2024-04-22T01:12:17.462323+0000","last_became_active":"2024-04-22T01:10:34.665474+0000","last_became_peered":"2024-04-22T01:10:34.665474+0000","last_unstale":"2024-04-22T01:12:17.462323+0000","last_undegraded":"2024-04-22T01:12:17.462323+0000","last_fullsized":"2024-04-22T01:12:17.462323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:01:53.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466996+0000","last_change":"2024-04-22T01:10:29.593722+0000","last_active":"2024-04-22T01:12:17.466996+0000","last_peered":"2024-04-22T01:12:17.466996+0000","last_clean":"2024-04-22T01:12:17.466996+0000","last_became_active":"2024-04-22T01:10:29.593405+0000","last_became_peered":"2024-04-22T01:10:29.593405+0000","last_unstale":"2024-04-22T01:12:17.466996+0000","last_undegraded":"2024-04-22T01:12:17.466996+0000","last_fullsized":"2024-04-22T01:12:17.466996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:47:41.684557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462862+0000","last_change":"2024-04-22T01:10:30.467503+0000","last_active":"2024-04-22T01:12:17.462862+0000","last_peered":"2024-04-22T01:12:17.462862+0000","last_clean":"2024-04-22T01:12:17.462862+0000","last_became_active":"2024-04-22T01:10:30.467046+0000","last_became_peered":"2024-04-22T01:10:30.467046+0000","last_unstale":"2024-04-22T01:12:17.462862+0000","last_undegraded":"2024-04-22T01:12:17.462862+0000","last_fullsized":"2024-04-22T01:12:17.462862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:23.219896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463203+0000","last_change":"2024-04-22T01:10:34.672933+0000","last_active":"2024-04-22T01:12:17.463203+0000","last_peered":"2024-04-22T01:12:17.463203+0000","last_clean":"2024-04-22T01:12:17.463203+0000","last_became_active":"2024-04-22T01:10:34.672411+0000","last_became_peered":"2024-04-22T01:10:34.672411+0000","last_unstale":"2024-04-22T01:12:17.463203+0000","last_undegraded":"2024-04-22T01:12:17.463203+0000","last_fullsized":"2024-04-22T01:12:17.463203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:26.303768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:43.721209+0000","last_change":"2024-04-22T01:10:29.591734+0000","last_active":"2024-04-22T01:12:43.721209+0000","last_peered":"2024-04-22T01:12:43.721209+0000","last_clean":"2024-04-22T01:12:43.721209+0000","last_became_active":"2024-04-22T01:10:29.590230+0000","last_became_peered":"2024-04-22T01:10:29.590230+0000","last_unstale":"2024-04-22T01:12:43.721209+0000","last_undegraded":"2024-04-22T01:12:43.721209+0000","last_fullsized":"2024-04-22T01:12:43.721209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:44.302772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.791049+0000","last_change":"2024-04-22T01:10:30.472284+0000","last_active":"2024-04-22T01:12:17.791049+0000","last_peered":"2024-04-22T01:12:17.791049+0000","last_clean":"2024-04-22T01:12:17.791049+0000","last_became_active":"2024-04-22T01:10:30.471926+0000","last_became_peered":"2024-04-22T01:10:30.471926+0000","last_unstale":"2024-04-22T01:12:17.791049+0000","last_undegraded":"2024-04-22T01:12:17.791049+0000","last_fullsized":"2024-04-22T01:12:17.791049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:57:06.058996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467241+0000","last_change":"2024-04-22T01:10:34.664782+0000","last_active":"2024-04-22T01:12:17.467241+0000","last_peered":"2024-04-22T01:12:17.467241+0000","last_clean":"2024-04-22T01:12:17.467241+0000","last_became_active":"2024-04-22T01:10:34.664365+0000","last_became_peered":"2024-04-22T01:10:34.664365+0000","last_unstale":"2024-04-22T01:12:17.467241+0000","last_undegraded":"2024-04-22T01:12:17.467241+0000","last_fullsized":"2024-04-22T01:12:17.467241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:55.989670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.792676+0000","last_change":"2024-04-22T01:12:17.791976+0000","last_active":"2024-04-22T01:12:17.792676+0000","last_peered":"2024-04-22T01:12:17.792676+0000","last_clean":"2024-04-22T01:12:17.792676+0000","last_became_active":"2024-04-22T01:12:17.791528+0000","last_became_peered":"2024-04-22T01:12:17.791528+0000","last_unstale":"2024-04-22T01:12:17.792676+0000","last_undegraded":"2024-04-22T01:12:17.792676+0000","last_fullsized":"2024-04-22T01:12:17.792676+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:24.398418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.466197+0000","last_change":"2024-04-22T01:10:30.469522+0000","last_active":"2024-04-22T01:12:17.466197+0000","last_peered":"2024-04-22T01:12:17.466197+0000","last_clean":"2024-04-22T01:12:17.466197+0000","last_became_active":"2024-04-22T01:10:30.469196+0000","last_became_peered":"2024-04-22T01:10:30.469196+0000","last_unstale":"2024-04-22T01:12:17.466197+0000","last_undegraded":"2024-04-22T01:12:17.466197+0000","last_fullsized":"2024-04-22T01:12:17.466197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:31:55.019717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462078+0000","last_change":"2024-04-22T01:10:34.668924+0000","last_active":"2024-04-22T01:12:17.462078+0000","last_peered":"2024-04-22T01:12:17.462078+0000","last_clean":"2024-04-22T01:12:17.462078+0000","last_became_active":"2024-04-22T01:10:34.668498+0000","last_became_peered":"2024-04-22T01:10:34.668498+0000","last_unstale":"2024-04-22T01:12:17.462078+0000","last_undegraded":"2024-04-22T01:12:17.462078+0000","last_fullsized":"2024-04-22T01:12:17.462078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:42.626059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:00.531275+0000","last_change":"2024-04-22T01:10:29.590662+0000","last_active":"2024-04-22T01:13:00.531275+0000","last_peered":"2024-04-22T01:13:00.531275+0000","last_clean":"2024-04-22T01:13:00.531275+0000","last_became_active":"2024-04-22T01:10:29.590022+0000","last_became_peered":"2024-04-22T01:10:29.590022+0000","last_unstale":"2024-04-22T01:13:00.531275+0000","last_undegraded":"2024-04-22T01:13:00.531275+0000","last_fullsized":"2024-04-22T01:13:00.531275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:38:43.635061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465714+0000","last_change":"2024-04-22T01:10:30.462005+0000","last_active":"2024-04-22T01:12:17.465714+0000","last_peered":"2024-04-22T01:12:17.465714+0000","last_clean":"2024-04-22T01:12:17.465714+0000","last_became_active":"2024-04-22T01:10:30.461716+0000","last_became_peered":"2024-04-22T01:10:30.461716+0000","last_unstale":"2024-04-22T01:12:17.465714+0000","last_undegraded":"2024-04-22T01:12:17.465714+0000","last_fullsized":"2024-04-22T01:12:17.465714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:52:03.777151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462218+0000","last_change":"2024-04-22T01:10:34.667787+0000","last_active":"2024-04-22T01:12:17.462218+0000","last_peered":"2024-04-22T01:12:17.462218+0000","last_clean":"2024-04-22T01:12:17.462218+0000","last_became_active":"2024-04-22T01:10:34.667296+0000","last_became_peered":"2024-04-22T01:10:34.667296+0000","last_unstale":"2024-04-22T01:12:17.462218+0000","last_undegraded":"2024-04-22T01:12:17.462218+0000","last_fullsized":"2024-04-22T01:12:17.462218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:04.501239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462941+0000","last_change":"2024-04-22T01:11:17.586033+0000","last_active":"2024-04-22T01:12:17.462941+0000","last_peered":"2024-04-22T01:12:17.462941+0000","last_clean":"2024-04-22T01:12:17.462941+0000","last_became_active":"2024-04-22T01:11:17.585529+0000","last_became_peered":"2024-04-22T01:11:17.585529+0000","last_unstale":"2024-04-22T01:12:17.462941+0000","last_undegraded":"2024-04-22T01:12:17.462941+0000","last_fullsized":"2024-04-22T01:12:17.462941+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:35:49.219761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:01.417087+0000","last_change":"2024-04-22T01:10:29.596651+0000","last_active":"2024-04-22T01:13:01.417087+0000","last_peered":"2024-04-22T01:13:01.417087+0000","last_clean":"2024-04-22T01:13:01.417087+0000","last_became_active":"2024-04-22T01:10:29.596255+0000","last_became_peered":"2024-04-22T01:10:29.596255+0000","last_unstale":"2024-04-22T01:13:01.417087+0000","last_undegraded":"2024-04-22T01:13:01.417087+0000","last_fullsized":"2024-04-22T01:13:01.417087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:51:58.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468114+0000","last_change":"2024-04-22T01:10:34.662724+0000","last_active":"2024-04-22T01:12:17.468114+0000","last_peered":"2024-04-22T01:12:17.468114+0000","last_clean":"2024-04-22T01:12:17.468114+0000","last_became_active":"2024-04-22T01:10:34.661374+0000","last_became_peered":"2024-04-22T01:10:34.661374+0000","last_unstale":"2024-04-22T01:12:17.468114+0000","last_undegraded":"2024-04-22T01:12:17.468114+0000","last_fullsized":"2024-04-22T01:12:17.468114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:03.154780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467519+0000","last_change":"2024-04-22T01:10:30.477008+0000","last_active":"2024-04-22T01:12:17.467519+0000","last_peered":"2024-04-22T01:12:17.467519+0000","last_clean":"2024-04-22T01:12:17.467519+0000","last_became_active":"2024-04-22T01:10:30.476742+0000","last_became_peered":"2024-04-22T01:10:30.476742+0000","last_unstale":"2024-04-22T01:12:17.467519+0000","last_undegraded":"2024-04-22T01:12:17.467519+0000","last_fullsized":"2024-04-22T01:12:17.467519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:28.356232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464425+0000","last_change":"2024-04-22T01:10:29.582159+0000","last_active":"2024-04-22T01:12:17.464425+0000","last_peered":"2024-04-22T01:12:17.464425+0000","last_clean":"2024-04-22T01:12:17.464425+0000","last_became_active":"2024-04-22T01:10:29.581792+0000","last_became_peered":"2024-04-22T01:10:29.581792+0000","last_unstale":"2024-04-22T01:12:17.464425+0000","last_undegraded":"2024-04-22T01:12:17.464425+0000","last_fullsized":"2024-04-22T01:12:17.464425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:24:20.943738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467202+0000","last_change":"2024-04-22T01:09:15.589460+0000","last_active":"2024-04-22T01:12:17.467202+0000","last_peered":"2024-04-22T01:12:17.467202+0000","last_clean":"2024-04-22T01:12:17.467202+0000","last_became_active":"2024-04-22T01:09:15.589006+0000","last_became_peered":"2024-04-22T01:09:15.589006+0000","last_unstale":"2024-04-22T01:12:17.467202+0000","last_undegraded":"2024-04-22T01:12:17.467202+0000","last_fullsized":"2024-04-22T01:12:17.467202+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:05:44.390182+0000","last_clean_scrub_stamp":"2024-04-22T01:05:44.390182+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:42.706006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463203+0000","last_change":"2024-04-22T01:10:34.662358+0000","last_active":"2024-04-22T01:12:17.463203+0000","last_peered":"2024-04-22T01:12:17.463203+0000","last_clean":"2024-04-22T01:12:17.463203+0000","last_became_active":"2024-04-22T01:10:34.661907+0000","last_became_peered":"2024-04-22T01:10:34.661907+0000","last_unstale":"2024-04-22T01:12:17.463203+0000","last_undegraded":"2024-04-22T01:12:17.463203+0000","last_fullsized":"2024-04-22T01:12:17.463203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:45:24.874555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.461817+0000","last_change":"2024-04-22T01:10:30.468037+0000","last_active":"2024-04-22T01:12:17.461817+0000","last_peered":"2024-04-22T01:12:17.461817+0000","last_clean":"2024-04-22T01:12:17.461817+0000","last_became_active":"2024-04-22T01:10:30.467491+0000","last_became_peered":"2024-04-22T01:10:30.467491+0000","last_unstale":"2024-04-22T01:12:17.461817+0000","last_undegraded":"2024-04-22T01:12:17.461817+0000","last_fullsized":"2024-04-22T01:12:17.461817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:45.647281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467945+0000","last_change":"2024-04-22T01:10:29.587112+0000","last_active":"2024-04-22T01:12:17.467945+0000","last_peered":"2024-04-22T01:12:17.467945+0000","last_clean":"2024-04-22T01:12:17.467945+0000","last_became_active":"2024-04-22T01:10:29.586780+0000","last_became_peered":"2024-04-22T01:10:29.586780+0000","last_unstale":"2024-04-22T01:12:17.467945+0000","last_undegraded":"2024-04-22T01:12:17.467945+0000","last_fullsized":"2024-04-22T01:12:17.467945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:06:49.364115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790799+0000","last_change":"2024-04-22T01:10:34.659354+0000","last_active":"2024-04-22T01:12:17.790799+0000","last_peered":"2024-04-22T01:12:17.790799+0000","last_clean":"2024-04-22T01:12:17.790799+0000","last_became_active":"2024-04-22T01:10:34.658898+0000","last_became_peered":"2024-04-22T01:10:34.658898+0000","last_unstale":"2024-04-22T01:12:17.790799+0000","last_undegraded":"2024-04-22T01:12:17.790799+0000","last_fullsized":"2024-04-22T01:12:17.790799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:44.304664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467511+0000","last_change":"2024-04-22T01:10:29.592672+0000","last_active":"2024-04-22T01:12:17.467511+0000","last_peered":"2024-04-22T01:12:17.467511+0000","last_clean":"2024-04-22T01:12:17.467511+0000","last_became_active":"2024-04-22T01:10:29.592368+0000","last_became_peered":"2024-04-22T01:10:29.592368+0000","last_unstale":"2024-04-22T01:12:17.467511+0000","last_undegraded":"2024-04-22T01:12:17.467511+0000","last_fullsized":"2024-04-22T01:12:17.467511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:32.609771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462204+0000","last_change":"2024-04-22T01:10:30.461994+0000","last_active":"2024-04-22T01:12:17.462204+0000","last_peered":"2024-04-22T01:12:17.462204+0000","last_clean":"2024-04-22T01:12:17.462204+0000","last_became_active":"2024-04-22T01:10:30.461392+0000","last_became_peered":"2024-04-22T01:10:30.461392+0000","last_unstale":"2024-04-22T01:12:17.462204+0000","last_undegraded":"2024-04-22T01:12:17.462204+0000","last_fullsized":"2024-04-22T01:12:17.462204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:04.575731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467391+0000","last_change":"2024-04-22T01:10:34.676412+0000","last_active":"2024-04-22T01:12:17.467391+0000","last_peered":"2024-04-22T01:12:17.467391+0000","last_clean":"2024-04-22T01:12:17.467391+0000","last_became_active":"2024-04-22T01:10:34.676034+0000","last_became_peered":"2024-04-22T01:10:34.676034+0000","last_unstale":"2024-04-22T01:12:17.467391+0000","last_undegraded":"2024-04-22T01:12:17.467391+0000","last_fullsized":"2024-04-22T01:12:17.467391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:03.669052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'12","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.797430+0000","last_change":"2024-04-22T01:12:17.796919+0000","last_active":"2024-04-22T01:12:17.797430+0000","last_peered":"2024-04-22T01:12:17.797430+0000","last_clean":"2024-04-22T01:12:17.797430+0000","last_became_active":"2024-04-22T01:12:17.791725+0000","last_became_peered":"2024-04-22T01:12:17.791725+0000","last_unstale":"2024-04-22T01:12:17.797430+0000","last_undegraded":"2024-04-22T01:12:17.797430+0000","last_fullsized":"2024-04-22T01:12:17.797430+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:50.607134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468273+0000","last_change":"2024-04-22T01:10:30.469485+0000","last_active":"2024-04-22T01:12:17.468273+0000","last_peered":"2024-04-22T01:12:17.468273+0000","last_clean":"2024-04-22T01:12:17.468273+0000","last_became_active":"2024-04-22T01:10:30.466745+0000","last_became_peered":"2024-04-22T01:10:30.466745+0000","last_unstale":"2024-04-22T01:12:17.468273+0000","last_undegraded":"2024-04-22T01:12:17.468273+0000","last_fullsized":"2024-04-22T01:12:17.468273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:00:01.083120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.469923+0000","last_change":"2024-04-22T01:10:34.665039+0000","last_active":"2024-04-22T01:12:17.469923+0000","last_peered":"2024-04-22T01:12:17.469923+0000","last_clean":"2024-04-22T01:12:17.469923+0000","last_became_active":"2024-04-22T01:10:34.664605+0000","last_became_peered":"2024-04-22T01:10:34.664605+0000","last_unstale":"2024-04-22T01:12:17.469923+0000","last_undegraded":"2024-04-22T01:12:17.469923+0000","last_fullsized":"2024-04-22T01:12:17.469923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:44.959158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.465115+0000","last_change":"2024-04-22T01:10:29.585709+0000","last_active":"2024-04-22T01:12:17.465115+0000","last_peered":"2024-04-22T01:12:17.465115+0000","last_clean":"2024-04-22T01:12:17.465115+0000","last_became_active":"2024-04-22T01:10:29.585381+0000","last_became_peered":"2024-04-22T01:10:29.585381+0000","last_unstale":"2024-04-22T01:12:17.465115+0000","last_undegraded":"2024-04-22T01:12:17.465115+0000","last_fullsized":"2024-04-22T01:12:17.465115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:15:53.677052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468452+0000","last_change":"2024-04-22T01:11:17.524897+0000","last_active":"2024-04-22T01:12:17.468452+0000","last_peered":"2024-04-22T01:12:17.468452+0000","last_clean":"2024-04-22T01:12:17.468452+0000","last_became_active":"2024-04-22T01:11:17.524423+0000","last_became_peered":"2024-04-22T01:11:17.524423+0000","last_unstale":"2024-04-22T01:12:17.468452+0000","last_undegraded":"2024-04-22T01:12:17.468452+0000","last_fullsized":"2024-04-22T01:12:17.468452+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:34:15.247122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464754+0000","last_change":"2024-04-22T01:10:34.671511+0000","last_active":"2024-04-22T01:12:17.464754+0000","last_peered":"2024-04-22T01:12:17.464754+0000","last_clean":"2024-04-22T01:12:17.464754+0000","last_became_active":"2024-04-22T01:10:34.670888+0000","last_became_peered":"2024-04-22T01:10:34.670888+0000","last_unstale":"2024-04-22T01:12:17.464754+0000","last_undegraded":"2024-04-22T01:12:17.464754+0000","last_fullsized":"2024-04-22T01:12:17.464754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:09:16.265695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470357+0000","last_change":"2024-04-22T01:10:29.588186+0000","last_active":"2024-04-22T01:12:17.470357+0000","last_peered":"2024-04-22T01:12:17.470357+0000","last_clean":"2024-04-22T01:12:17.470357+0000","last_became_active":"2024-04-22T01:10:29.587856+0000","last_became_peered":"2024-04-22T01:10:29.587856+0000","last_unstale":"2024-04-22T01:12:17.470357+0000","last_undegraded":"2024-04-22T01:12:17.470357+0000","last_fullsized":"2024-04-22T01:12:17.470357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:10.707233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467064+0000","last_change":"2024-04-22T01:10:30.471777+0000","last_active":"2024-04-22T01:12:17.467064+0000","last_peered":"2024-04-22T01:12:17.467064+0000","last_clean":"2024-04-22T01:12:17.467064+0000","last_became_active":"2024-04-22T01:10:30.471398+0000","last_became_peered":"2024-04-22T01:10:30.471398+0000","last_unstale":"2024-04-22T01:12:17.467064+0000","last_undegraded":"2024-04-22T01:12:17.467064+0000","last_fullsized":"2024-04-22T01:12:17.467064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:51.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.471987+0000","last_change":"2024-04-22T01:12:17.471987+0000","last_active":"2024-04-22T01:12:17.471987+0000","last_peered":"2024-04-22T01:12:17.471987+0000","last_clean":"2024-04-22T01:12:17.471987+0000","last_became_active":"2024-04-22T01:12:17.471369+0000","last_became_peered":"2024-04-22T01:12:17.471369+0000","last_unstale":"2024-04-22T01:12:17.471987+0000","last_undegraded":"2024-04-22T01:12:17.471987+0000","last_fullsized":"2024-04-22T01:12:17.471987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:29:20.856968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,8],"acting":[10,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467697+0000","last_change":"2024-04-22T01:10:29.587649+0000","last_active":"2024-04-22T01:12:17.467697+0000","last_peered":"2024-04-22T01:12:17.467697+0000","last_clean":"2024-04-22T01:12:17.467697+0000","last_became_active":"2024-04-22T01:10:29.587297+0000","last_became_peered":"2024-04-22T01:10:29.587297+0000","last_unstale":"2024-04-22T01:12:17.467697+0000","last_undegraded":"2024-04-22T01:12:17.467697+0000","last_fullsized":"2024-04-22T01:12:17.467697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:38:11.853825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790341+0000","last_change":"2024-04-22T01:10:30.461399+0000","last_active":"2024-04-22T01:12:17.790341+0000","last_peered":"2024-04-22T01:12:17.790341+0000","last_clean":"2024-04-22T01:12:17.790341+0000","last_became_active":"2024-04-22T01:10:30.461122+0000","last_became_peered":"2024-04-22T01:10:30.461122+0000","last_unstale":"2024-04-22T01:12:17.790341+0000","last_undegraded":"2024-04-22T01:12:17.790341+0000","last_fullsized":"2024-04-22T01:12:17.790341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:33.169504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464549+0000","last_change":"2024-04-22T01:10:34.676810+0000","last_active":"2024-04-22T01:12:17.464549+0000","last_peered":"2024-04-22T01:12:17.464549+0000","last_clean":"2024-04-22T01:12:17.464549+0000","last_became_active":"2024-04-22T01:10:34.676299+0000","last_became_peered":"2024-04-22T01:10:34.676299+0000","last_unstale":"2024-04-22T01:12:17.464549+0000","last_undegraded":"2024-04-22T01:12:17.464549+0000","last_fullsized":"2024-04-22T01:12:17.464549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:32:52.919529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:38.991705+0000","last_change":"2024-04-22T01:10:29.589833+0000","last_active":"2024-04-22T01:12:38.991705+0000","last_peered":"2024-04-22T01:12:38.991705+0000","last_clean":"2024-04-22T01:12:38.991705+0000","last_became_active":"2024-04-22T01:10:29.589515+0000","last_became_peered":"2024-04-22T01:10:29.589515+0000","last_unstale":"2024-04-22T01:12:38.991705+0000","last_undegraded":"2024-04-22T01:12:38.991705+0000","last_fullsized":"2024-04-22T01:12:38.991705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:50.349464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467870+0000","last_change":"2024-04-22T01:10:30.471954+0000","last_active":"2024-04-22T01:12:17.467870+0000","last_peered":"2024-04-22T01:12:17.467870+0000","last_clean":"2024-04-22T01:12:17.467870+0000","last_became_active":"2024-04-22T01:10:30.469598+0000","last_became_peered":"2024-04-22T01:10:30.469598+0000","last_unstale":"2024-04-22T01:12:17.467870+0000","last_undegraded":"2024-04-22T01:12:17.467870+0000","last_fullsized":"2024-04-22T01:12:17.467870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:20.742880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.464696+0000","last_change":"2024-04-22T01:10:34.673246+0000","last_active":"2024-04-22T01:12:17.464696+0000","last_peered":"2024-04-22T01:12:17.464696+0000","last_clean":"2024-04-22T01:12:17.464696+0000","last_became_active":"2024-04-22T01:10:34.672796+0000","last_became_peered":"2024-04-22T01:10:34.672796+0000","last_unstale":"2024-04-22T01:12:17.464696+0000","last_undegraded":"2024-04-22T01:12:17.464696+0000","last_fullsized":"2024-04-22T01:12:17.464696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:21:17.797425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467236+0000","last_change":"2024-04-22T01:10:29.594040+0000","last_active":"2024-04-22T01:12:17.467236+0000","last_peered":"2024-04-22T01:12:17.467236+0000","last_clean":"2024-04-22T01:12:17.467236+0000","last_became_active":"2024-04-22T01:10:29.593722+0000","last_became_peered":"2024-04-22T01:10:29.593722+0000","last_unstale":"2024-04-22T01:12:17.467236+0000","last_undegraded":"2024-04-22T01:12:17.467236+0000","last_fullsized":"2024-04-22T01:12:17.467236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:25.833882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468077+0000","last_change":"2024-04-22T01:10:30.466195+0000","last_active":"2024-04-22T01:12:17.468077+0000","last_peered":"2024-04-22T01:12:17.468077+0000","last_clean":"2024-04-22T01:12:17.468077+0000","last_became_active":"2024-04-22T01:10:30.465437+0000","last_became_peered":"2024-04-22T01:10:30.465437+0000","last_unstale":"2024-04-22T01:12:17.468077+0000","last_undegraded":"2024-04-22T01:12:17.468077+0000","last_fullsized":"2024-04-22T01:12:17.468077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:45:21.245134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470610+0000","last_change":"2024-04-22T01:10:34.661068+0000","last_active":"2024-04-22T01:12:17.470610+0000","last_peered":"2024-04-22T01:12:17.470610+0000","last_clean":"2024-04-22T01:12:17.470610+0000","last_became_active":"2024-04-22T01:10:34.660603+0000","last_became_peered":"2024-04-22T01:10:34.660603+0000","last_unstale":"2024-04-22T01:12:17.470610+0000","last_undegraded":"2024-04-22T01:12:17.470610+0000","last_fullsized":"2024-04-22T01:12:17.470610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:57.921779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:13:03.624385+0000","last_change":"2024-04-22T01:10:29.586866+0000","last_active":"2024-04-22T01:13:03.624385+0000","last_peered":"2024-04-22T01:13:03.624385+0000","last_clean":"2024-04-22T01:13:03.624385+0000","last_became_active":"2024-04-22T01:10:29.586556+0000","last_became_peered":"2024-04-22T01:10:29.586556+0000","last_unstale":"2024-04-22T01:13:03.624385+0000","last_undegraded":"2024-04-22T01:13:03.624385+0000","last_fullsized":"2024-04-22T01:13:03.624385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:37:41.132733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2093,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468326+0000","last_change":"2024-04-22T01:10:30.477253+0000","last_active":"2024-04-22T01:12:17.468326+0000","last_peered":"2024-04-22T01:12:17.468326+0000","last_clean":"2024-04-22T01:12:17.468326+0000","last_became_active":"2024-04-22T01:10:30.476957+0000","last_became_peered":"2024-04-22T01:10:30.476957+0000","last_unstale":"2024-04-22T01:12:17.468326+0000","last_undegraded":"2024-04-22T01:12:17.468326+0000","last_fullsized":"2024-04-22T01:12:17.468326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:16:06.156257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467471+0000","last_change":"2024-04-22T01:10:34.677909+0000","last_active":"2024-04-22T01:12:17.467471+0000","last_peered":"2024-04-22T01:12:17.467471+0000","last_clean":"2024-04-22T01:12:17.467471+0000","last_became_active":"2024-04-22T01:10:34.677398+0000","last_became_peered":"2024-04-22T01:10:34.677398+0000","last_unstale":"2024-04-22T01:12:17.467471+0000","last_undegraded":"2024-04-22T01:12:17.467471+0000","last_fullsized":"2024-04-22T01:12:17.467471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:22:25.191622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.463429+0000","last_change":"2024-04-22T01:10:29.600101+0000","last_active":"2024-04-22T01:12:17.463429+0000","last_peered":"2024-04-22T01:12:17.463429+0000","last_clean":"2024-04-22T01:12:17.463429+0000","last_became_active":"2024-04-22T01:10:29.599741+0000","last_became_peered":"2024-04-22T01:10:29.599741+0000","last_unstale":"2024-04-22T01:12:17.463429+0000","last_undegraded":"2024-04-22T01:12:17.463429+0000","last_fullsized":"2024-04-22T01:12:17.463429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:37:52.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467776+0000","last_change":"2024-04-22T01:11:17.313900+0000","last_active":"2024-04-22T01:12:17.467776+0000","last_peered":"2024-04-22T01:12:17.467776+0000","last_clean":"2024-04-22T01:12:17.467776+0000","last_became_active":"2024-04-22T01:11:17.313493+0000","last_became_peered":"2024-04-22T01:11:17.313493+0000","last_unstale":"2024-04-22T01:12:17.467776+0000","last_undegraded":"2024-04-22T01:12:17.467776+0000","last_fullsized":"2024-04-22T01:12:17.467776+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:30:42.909373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462618+0000","last_change":"2024-04-22T01:10:34.665214+0000","last_active":"2024-04-22T01:12:17.462618+0000","last_peered":"2024-04-22T01:12:17.462618+0000","last_clean":"2024-04-22T01:12:17.462618+0000","last_became_active":"2024-04-22T01:10:34.664771+0000","last_became_peered":"2024-04-22T01:10:34.664771+0000","last_unstale":"2024-04-22T01:12:17.462618+0000","last_undegraded":"2024-04-22T01:12:17.462618+0000","last_fullsized":"2024-04-22T01:12:17.462618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:33.620107+0000","last_clean_scrub_stamp":"2024-04-22T01:10:33.620107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:44:36.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462388+0000","last_change":"2024-04-22T01:10:29.597867+0000","last_active":"2024-04-22T01:12:17.462388+0000","last_peered":"2024-04-22T01:12:17.462388+0000","last_clean":"2024-04-22T01:12:17.462388+0000","last_became_active":"2024-04-22T01:10:29.597502+0000","last_became_peered":"2024-04-22T01:10:29.597502+0000","last_unstale":"2024-04-22T01:12:17.462388+0000","last_undegraded":"2024-04-22T01:12:17.462388+0000","last_fullsized":"2024-04-22T01:12:17.462388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:28.558465+0000","last_clean_scrub_stamp":"2024-04-22T01:10:28.558465+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:45.713841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468003+0000","last_change":"2024-04-22T01:10:30.469969+0000","last_active":"2024-04-22T01:12:17.468003+0000","last_peered":"2024-04-22T01:12:17.468003+0000","last_clean":"2024-04-22T01:12:17.468003+0000","last_became_active":"2024-04-22T01:10:30.469657+0000","last_became_peered":"2024-04-22T01:10:30.469657+0000","last_unstale":"2024-04-22T01:12:17.468003+0000","last_undegraded":"2024-04-22T01:12:17.468003+0000","last_fullsized":"2024-04-22T01:12:17.468003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:01:32.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790421+0000","last_change":"2024-04-22T01:10:30.460764+0000","last_active":"2024-04-22T01:12:17.790421+0000","last_peered":"2024-04-22T01:12:17.790421+0000","last_clean":"2024-04-22T01:12:17.790421+0000","last_became_active":"2024-04-22T01:10:30.460372+0000","last_became_peered":"2024-04-22T01:10:30.460372+0000","last_unstale":"2024-04-22T01:12:17.790421+0000","last_undegraded":"2024-04-22T01:12:17.790421+0000","last_fullsized":"2024-04-22T01:12:17.790421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:19:00.625595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470087+0000","last_change":"2024-04-22T01:10:30.470679+0000","last_active":"2024-04-22T01:12:17.470087+0000","last_peered":"2024-04-22T01:12:17.470087+0000","last_clean":"2024-04-22T01:12:17.470087+0000","last_became_active":"2024-04-22T01:10:30.470355+0000","last_became_peered":"2024-04-22T01:10:30.470355+0000","last_unstale":"2024-04-22T01:12:17.470087+0000","last_undegraded":"2024-04-22T01:12:17.470087+0000","last_fullsized":"2024-04-22T01:12:17.470087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:40:34.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470224+0000","last_change":"2024-04-22T01:10:30.469445+0000","last_active":"2024-04-22T01:12:17.470224+0000","last_peered":"2024-04-22T01:12:17.470224+0000","last_clean":"2024-04-22T01:12:17.470224+0000","last_became_active":"2024-04-22T01:10:30.469135+0000","last_became_peered":"2024-04-22T01:10:30.469135+0000","last_unstale":"2024-04-22T01:12:17.470224+0000","last_undegraded":"2024-04-22T01:12:17.470224+0000","last_fullsized":"2024-04-22T01:12:17.470224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:18:54.246612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467790+0000","last_change":"2024-04-22T01:10:30.476402+0000","last_active":"2024-04-22T01:12:17.467790+0000","last_peered":"2024-04-22T01:12:17.467790+0000","last_clean":"2024-04-22T01:12:17.467790+0000","last_became_active":"2024-04-22T01:10:30.476037+0000","last_became_peered":"2024-04-22T01:10:30.476037+0000","last_unstale":"2024-04-22T01:12:17.467790+0000","last_undegraded":"2024-04-22T01:12:17.467790+0000","last_fullsized":"2024-04-22T01:12:17.467790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:21:42.897348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462189+0000","last_change":"2024-04-22T01:10:30.468255+0000","last_active":"2024-04-22T01:12:17.462189+0000","last_peered":"2024-04-22T01:12:17.462189+0000","last_clean":"2024-04-22T01:12:17.462189+0000","last_became_active":"2024-04-22T01:10:30.467854+0000","last_became_peered":"2024-04-22T01:10:30.467854+0000","last_unstale":"2024-04-22T01:12:17.462189+0000","last_undegraded":"2024-04-22T01:12:17.462189+0000","last_fullsized":"2024-04-22T01:12:17.462189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:47:20.412202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790840+0000","last_change":"2024-04-22T01:10:30.465511+0000","last_active":"2024-04-22T01:12:17.790840+0000","last_peered":"2024-04-22T01:12:17.790840+0000","last_clean":"2024-04-22T01:12:17.790840+0000","last_became_active":"2024-04-22T01:10:30.465208+0000","last_became_peered":"2024-04-22T01:10:30.465208+0000","last_unstale":"2024-04-22T01:12:17.790840+0000","last_undegraded":"2024-04-22T01:12:17.790840+0000","last_fullsized":"2024-04-22T01:12:17.790840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:20.662304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470692+0000","last_change":"2024-04-22T01:10:30.467305+0000","last_active":"2024-04-22T01:12:17.470692+0000","last_peered":"2024-04-22T01:12:17.470692+0000","last_clean":"2024-04-22T01:12:17.470692+0000","last_became_active":"2024-04-22T01:10:30.466959+0000","last_became_peered":"2024-04-22T01:10:30.466959+0000","last_unstale":"2024-04-22T01:12:17.470692+0000","last_undegraded":"2024-04-22T01:12:17.470692+0000","last_fullsized":"2024-04-22T01:12:17.470692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:28:58.082475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.468527+0000","last_change":"2024-04-22T01:10:30.474769+0000","last_active":"2024-04-22T01:12:17.468527+0000","last_peered":"2024-04-22T01:12:17.468527+0000","last_clean":"2024-04-22T01:12:17.468527+0000","last_became_active":"2024-04-22T01:10:30.474398+0000","last_became_peered":"2024-04-22T01:10:30.474398+0000","last_unstale":"2024-04-22T01:12:17.468527+0000","last_undegraded":"2024-04-22T01:12:17.468527+0000","last_fullsized":"2024-04-22T01:12:17.468527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:14:10.255551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462784+0000","last_change":"2024-04-22T01:10:30.469672+0000","last_active":"2024-04-22T01:12:17.462784+0000","last_peered":"2024-04-22T01:12:17.462784+0000","last_clean":"2024-04-22T01:12:17.462784+0000","last_became_active":"2024-04-22T01:10:30.468499+0000","last_became_peered":"2024-04-22T01:10:30.468499+0000","last_unstale":"2024-04-22T01:12:17.462784+0000","last_undegraded":"2024-04-22T01:12:17.462784+0000","last_fullsized":"2024-04-22T01:12:17.462784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:06:55.835229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.467872+0000","last_change":"2024-04-22T01:10:30.466279+0000","last_active":"2024-04-22T01:12:17.467872+0000","last_peered":"2024-04-22T01:12:17.467872+0000","last_clean":"2024-04-22T01:12:17.467872+0000","last_became_active":"2024-04-22T01:10:30.465964+0000","last_became_peered":"2024-04-22T01:10:30.465964+0000","last_unstale":"2024-04-22T01:12:17.467872+0000","last_undegraded":"2024-04-22T01:12:17.467872+0000","last_fullsized":"2024-04-22T01:12:17.467872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:05.527717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.470799+0000","last_change":"2024-04-22T01:11:17.586313+0000","last_active":"2024-04-22T01:12:17.470799+0000","last_peered":"2024-04-22T01:12:17.470799+0000","last_clean":"2024-04-22T01:12:17.470799+0000","last_became_active":"2024-04-22T01:11:17.585705+0000","last_became_peered":"2024-04-22T01:11:17.585705+0000","last_unstale":"2024-04-22T01:12:17.470799+0000","last_undegraded":"2024-04-22T01:12:17.470799+0000","last_fullsized":"2024-04-22T01:12:17.470799+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:36:02.313774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.462485+0000","last_change":"2024-04-22T01:10:30.465333+0000","last_active":"2024-04-22T01:12:17.462485+0000","last_peered":"2024-04-22T01:12:17.462485+0000","last_clean":"2024-04-22T01:12:17.462485+0000","last_became_active":"2024-04-22T01:10:30.464987+0000","last_became_peered":"2024-04-22T01:10:30.464987+0000","last_unstale":"2024-04-22T01:12:17.462485+0000","last_undegraded":"2024-04-22T01:12:17.462485+0000","last_fullsized":"2024-04-22T01:12:17.462485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:39.619973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-22T01:12:17.790617+0000","last_change":"2024-04-22T01:10:30.467037+0000","last_active":"2024-04-22T01:12:17.790617+0000","last_peered":"2024-04-22T01:12:17.790617+0000","last_clean":"2024-04-22T01:12:17.790617+0000","last_became_active":"2024-04-22T01:10:30.466714+0000","last_became_peered":"2024-04-22T01:10:30.466714+0000","last_unstale":"2024-04-22T01:12:17.790617+0000","last_undegraded":"2024-04-22T01:12:17.790617+0000","last_fullsized":"2024-04-22T01:12:17.790617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:10:29.563749+0000","last_clean_scrub_stamp":"2024-04-22T01:10:29.563749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:29.812044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286399,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69775,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1256,"num_read_kb":700,"num_write":257,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":672,"num_keys_recovered":7,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":217953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":521373,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1175552,"data_stored":1168157,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":524476,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":520086,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1117704,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495687,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":515884,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":583490,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":517872,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":584884,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":515592,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":519958,"data_compressed":26606,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1113369,"data_compressed":63019,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34778,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36413,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T01:13:07.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/1189444601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54103340 msgr2=0x7f2f54109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/1189444601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54103340 0x7f2f54109bd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c009ca0 tx=0x7f2f4c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/1189444601 shutdown_connections 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/1189444601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54103340 0x7f2f54109bd0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/1189444601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f54102980 0x7f2f54102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/1189444601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f54101790 0x7f2f54101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/1189444601 >> 172.21.15.38:0/1189444601 conn(0x7f2f540fcd20 msgr2=0x7f2f540ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/1189444601 shutdown_connections 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.849+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/1189444601 wait complete. 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 Processor -- start 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 -- start start 2024-04-22T01:13:07.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f54101790 0x7f2f54197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 0x7f2f54197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f54103340 0x7f2f5419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5410d3f0 con 0x7f2f54102980 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5410d270 con 0x7f2f54101790 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5b9bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5410d570 con 0x7f2f54103340 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f58f5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 0x7f2f54197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f58f5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 0x7f2f54197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37344/0 (socket says 172.21.15.38:37344) 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f58f5a640 1 -- 172.21.15.38:0/3088330195 learned_addr learned my addr 172.21.15.38:0/3088330195 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f59f5c640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f54103340 0x7f2f5419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.850+0000 7f2f5975b640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f54101790 0x7f2f54197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f58f5a640 1 -- 172.21.15.38:0/3088330195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f54103340 msgr2=0x7f2f5419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f58f5a640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f54103340 0x7f2f5419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f58f5a640 1 -- 172.21.15.38:0/3088330195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f54101790 msgr2=0x7f2f54197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f58f5a640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f54101790 0x7f2f54197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:07.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f58f5a640 1 -- 172.21.15.38:0/3088330195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5419f6c0 con 0x7f2f54102980 2024-04-22T01:13:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f58f5a640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 0x7f2f54197fe0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e9f0 tx=0x7f2f4400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4400cd80 con 0x7f2f54102980 2024-04-22T01:13:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f44010910 con 0x7f2f54102980 2024-04-22T01:13:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5419f9b0 con 0x7f2f54102980 2024-04-22T01:13:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44018990 con 0x7f2f54102980 2024-04-22T01:13:07.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.851+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f5419ff00 con 0x7f2f54102980 2024-04-22T01:13:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.852+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f54108a50 con 0x7f2f54102980 2024-04-22T01:13:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.853+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f2f44010ac0 con 0x7f2f54102980 2024-04-22T01:13:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.854+0000 7f2f4a7fc640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2f400a2c20 0x7f2f400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.854+0000 7f2f5975b640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2f400a2c20 0x7f2f400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.854+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f2f440cf870 con 0x7f2f54102980 2024-04-22T01:13:07.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.854+0000 7f2f5975b640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2f400a2c20 0x7f2f400a50e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005fd0 tx=0x7f2f5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:07.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:07.858+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2f4409bc10 con 0x7f2f54102980 2024-04-22T01:13:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:07 smithi203 ceph-mon[31825]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-22T01:13:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:07 smithi203 ceph-mon[31825]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:07 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:07 smithi203 ceph-mon[31825]: from='client.25096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:07 smithi203 ceph-mon[31825]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.045+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f5402d370 con 0x7f2f54102980 2024-04-22T01:13:08.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.047+0000 7f2f4a7fc640 1 -- 172.21.15.38:0/3088330195 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7f2f4409ff10 con 0x7f2f54102980 2024-04-22T01:13:08.046 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:08.046 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:13:08.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2f400a2c20 msgr2=0x7f2f400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2f400a2c20 0x7f2f400a50e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005fd0 tx=0x7f2f5000b410 comp rx=0 tx=0).stop 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 msgr2=0x7f2f54197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 0x7f2f54197fe0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e9f0 tx=0x7f2f4400eeb0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 shutdown_connections 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2f400a2c20 0x7f2f400a50e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f54103340 0x7f2f5419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f54102980 0x7f2f54197fe0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 --2- 172.21.15.38:0/3088330195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f54101790 0x7f2f54197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 >> 172.21.15.38:0/3088330195 conn(0x7f2f540fcd20 msgr2=0x7f2f54105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 shutdown_connections 2024-04-22T01:13:08.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.049+0000 7f2f5b9bf640 1 -- 172.21.15.38:0/3088330195 wait complete. 2024-04-22T01:13:08.052 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:07 smithi038 ceph-mon[28043]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-22T01:13:08.052 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:07 smithi038 ceph-mon[28043]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.052 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:08.052 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:07 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:08.052 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:07 smithi038 ceph-mon[28043]: from='client.25096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.052 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:07 smithi038 ceph-mon[28043]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.061 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:13:08.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:13:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:07 smithi111 ceph-mon[31057]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-22T01:13:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:07 smithi111 ceph-mon[31057]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:07 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:07 smithi111 ceph-mon[31057]: from='client.25096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:07 smithi111 ceph-mon[31057]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T01:13:08.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.171+0000 7f87c5640640 1 -- 172.21.15.38:0/1255977184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0104b30 msgr2=0x7f87c0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.171+0000 7f87c5640640 1 --2- 172.21.15.38:0/1255977184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0104b30 0x7f87c0106f20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009900 tx=0x7f87a802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:08.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 -- 172.21.15.38:0/1255977184 shutdown_connections 2024-04-22T01:13:08.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 --2- 172.21.15.38:0/1255977184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0107460 0x7f87c01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 --2- 172.21.15.38:0/1255977184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0104b30 0x7f87c0106f20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 --2- 172.21.15.38:0/1255977184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87c0102200 0x7f87c01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 -- 172.21.15.38:0/1255977184 >> 172.21.15.38:0/1255977184 conn(0x7f87c00fbbf0 msgr2=0x7f87c00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:08.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 -- 172.21.15.38:0/1255977184 shutdown_connections 2024-04-22T01:13:08.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 -- 172.21.15.38:0/1255977184 wait complete. 2024-04-22T01:13:08.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 Processor -- start 2024-04-22T01:13:08.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 -- start start 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 0x7f87c0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.172+0000 7f87c5640640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0104b30 0x7f87c0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87c5640640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87c0107460 0x7f87c019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87c5640640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87c010d410 con 0x7f87c0102200 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87c5640640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f87c010d290 con 0x7f87c0104b30 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87c5640640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87c010d590 con 0x7f87c0107460 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 0x7f87c0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 0x7f87c0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37352/0 (socket says 172.21.15.38:37352) 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 -- 172.21.15.38:0/3262273692 learned_addr learned my addr 172.21.15.38:0/3262273692 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87b7fff640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0104b30 0x7f87c0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87bf7fe640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87c0107460 0x7f87c019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 -- 172.21.15.38:0/3262273692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87c0107460 msgr2=0x7f87c019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87c0107460 0x7f87c019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 -- 172.21.15.38:0/3262273692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0104b30 msgr2=0x7f87c0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0104b30 0x7f87c0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 -- 172.21.15.38:0/3262273692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c019f730 con 0x7f87c0102200 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87b7fff640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0104b30 0x7f87c0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87beffd640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 0x7f87c0197b10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f87b000e9f0 tx=0x7f87b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b000cd80 con 0x7f87c0102200 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c019fa20 con 0x7f87c0102200 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.173+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b0005500 con 0x7f87c0102200 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.174+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0010690 con 0x7f87c0102200 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.174+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c019fef0 con 0x7f87c0102200 2024-04-22T01:13:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.175+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c00fd6e0 con 0x7f87c0102200 2024-04-22T01:13:08.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.176+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f87b00046c0 con 0x7f87c0102200 2024-04-22T01:13:08.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.177+0000 7f87bcff9640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87ac0a2c20 0x7f87ac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.177+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f87b00cf9c0 con 0x7f87c0102200 2024-04-22T01:13:08.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.177+0000 7f87b7fff640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87ac0a2c20 0x7f87ac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.178+0000 7f87b7fff640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87ac0a2c20 0x7f87ac0a50e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f87a8000c00 tx=0x7f87a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:08.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.182+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f87b009c3b0 con 0x7f87c0102200 2024-04-22T01:13:08.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.368+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87c002d170 con 0x7f87c0102200 2024-04-22T01:13:08.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.368+0000 7f87bcff9640 1 -- 172.21.15.38:0/3262273692 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f87c002d170 con 0x7f87c0102200 2024-04-22T01:13:08.368 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87ac0a2c20 msgr2=0x7f87ac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87ac0a2c20 0x7f87ac0a50e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f87a8000c00 tx=0x7f87a8005fd0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 msgr2=0x7f87c0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 0x7f87c0197b10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f87b000e9f0 tx=0x7f87b000edf0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 shutdown_connections 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87ac0a2c20 0x7f87ac0a50e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87c0107460 0x7f87c019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c0104b30 0x7f87c0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 --2- 172.21.15.38:0/3262273692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87c0102200 0x7f87c0197b10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 >> 172.21.15.38:0/3262273692 conn(0x7f87c00fbbf0 msgr2=0x7f87c0102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:08.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 shutdown_connections 2024-04-22T01:13:08.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.370+0000 7f87c5640640 1 -- 172.21.15.38:0/3262273692 wait complete. 2024-04-22T01:13:08.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:13:08.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 -- 172.21.15.38:0/2660031233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40103350 msgr2=0x7f7a40109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 --2- 172.21.15.38:0/2660031233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40103350 0x7f7a40109be0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009ca0 tx=0x7f7a3802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 -- 172.21.15.38:0/2660031233 shutdown_connections 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 --2- 172.21.15.38:0/2660031233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40103350 0x7f7a40109be0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 --2- 172.21.15.38:0/2660031233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a40102990 0x7f7a40102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 --2- 172.21.15.38:0/2660031233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a401017a0 0x7f7a40101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 -- 172.21.15.38:0/2660031233 >> 172.21.15.38:0/2660031233 conn(0x7f7a400fcd10 msgr2=0x7f7a400ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 -- 172.21.15.38:0/2660031233 shutdown_connections 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.489+0000 7f7a47725640 1 -- 172.21.15.38:0/2660031233 wait complete. 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 Processor -- start 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 -- start start 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a401017a0 0x7f7a40197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 0x7f7a40197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a40103350 0x7f7a4019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4010d420 con 0x7f7a40102990 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4010d2a0 con 0x7f7a401017a0 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a47725640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4010d5a0 con 0x7f7a40103350 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a44cc0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 0x7f7a40197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a454c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a401017a0 0x7f7a40197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a44cc0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 0x7f7a40197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37374/0 (socket says 172.21.15.38:37374) 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a454c1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a401017a0 0x7f7a40197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43602/0 (socket says 172.21.15.38:43602) 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a44cc0640 1 -- 172.21.15.38:0/700476028 learned_addr learned my addr 172.21.15.38:0/700476028 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a45cc2640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a40103350 0x7f7a4019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a45cc2640 1 -- 172.21.15.38:0/700476028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a401017a0 msgr2=0x7f7a40197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a45cc2640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a401017a0 0x7f7a40197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a45cc2640 1 -- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 msgr2=0x7f7a40197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a45cc2640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 0x7f7a40197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.490+0000 7f7a45cc2640 1 -- 172.21.15.38:0/700476028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4019f680 con 0x7f7a40103350 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a44cc0640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 0x7f7a40197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:08.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a45cc2640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a40103350 0x7f7a4019f010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a38037040 tx=0x7f7a38032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:08.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38041020 con 0x7f7a40103350 2024-04-22T01:13:08.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4019f910 con 0x7f7a40103350 2024-04-22T01:13:08.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a4019fe20 con 0x7f7a40103350 2024-04-22T01:13:08.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a38034cd0 con 0x7f7a40103350 2024-04-22T01:13:08.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.491+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38039040 con 0x7f7a40103350 2024-04-22T01:13:08.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.492+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a40108ac0 con 0x7f7a40103350 2024-04-22T01:13:08.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.493+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f7a38044020 con 0x7f7a40103350 2024-04-22T01:13:08.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.494+0000 7f7a367fc640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7a2c0a2c20 0x7f7a2c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.494+0000 7f7a454c1640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7a2c0a2c20 0x7f7a2c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.494+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7a380f1700 con 0x7f7a40103350 2024-04-22T01:13:08.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.494+0000 7f7a454c1640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7a2c0a2c20 0x7f7a2c0a50e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7a401027f0 tx=0x7f7a3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:08.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.497+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7a3804b020 con 0x7f7a40103350 2024-04-22T01:13:08.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.716+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a4002d170 con 0x7f7a40103350 2024-04-22T01:13:08.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.717+0000 7f7a367fc640 1 -- 172.21.15.38:0/700476028 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f7a380be240 con 0x7f7a40103350 2024-04-22T01:13:08.716 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7a2c0a2c20 msgr2=0x7f7a2c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7a2c0a2c20 0x7f7a2c0a50e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7a401027f0 tx=0x7f7a3c005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a40103350 msgr2=0x7f7a4019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a40103350 0x7f7a4019f010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a38037040 tx=0x7f7a38032ea0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 shutdown_connections 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7a2c0a2c20 0x7f7a2c0a50e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a40103350 0x7f7a4019f010 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a40102990 0x7f7a40197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 --2- 172.21.15.38:0/700476028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a401017a0 0x7f7a40197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 >> 172.21.15.38:0/700476028 conn(0x7f7a400fcd10 msgr2=0x7f7a40105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:08.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 shutdown_connections 2024-04-22T01:13:08.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.719+0000 7f7a47725640 1 -- 172.21.15.38:0/700476028 wait complete. 2024-04-22T01:13:08.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1339572521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 msgr2=0x7fe9281045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1339572521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe9281045f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe918009900 tx=0x7fe91802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1339572521 shutdown_connections 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1339572521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe928107460 0x7fe9281098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1339572521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928104b30 0x7fe928106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1339572521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe9281045f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1339572521 >> 172.21.15.38:0/1339572521 conn(0x7fe9280fbbf0 msgr2=0x7fe9280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1339572521 shutdown_connections 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.844+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1339572521 wait complete. 2024-04-22T01:13:08.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 Processor -- start 2024-04-22T01:13:08.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 -- start start 2024-04-22T01:13:08.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe928197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe928104b30 0x7fe928198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928107460 0x7fe92819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe92810d450 con 0x7fe928102200 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe92810d2d0 con 0x7fe928107460 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe92d8c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe92810d5d0 con 0x7fe928104b30 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe926ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe928197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe926ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe928197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37384/0 (socket says 172.21.15.38:37384) 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe926ffd640 1 -- 172.21.15.38:0/1145403390 learned_addr learned my addr 172.21.15.38:0/1145403390 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe9277fe640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928107460 0x7fe92819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.845+0000 7fe9267fc640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe928104b30 0x7fe928198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe9277fe640 1 -- 172.21.15.38:0/1145403390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe928104b30 msgr2=0x7fe928198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe9277fe640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe928104b30 0x7fe928198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe9277fe640 1 -- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 msgr2=0x7fe928197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe9277fe640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe928197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe9277fe640 1 -- 172.21.15.38:0/1145403390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92819f750 con 0x7fe928107460 2024-04-22T01:13:08.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe9277fe640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928107460 0x7fe92819f0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe92000da10 tx=0x7fe92000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:08.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92000bd70 con 0x7fe928107460 2024-04-22T01:13:08.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe920003e50 con 0x7fe928107460 2024-04-22T01:13:08.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92819fa40 con 0x7fe928107460 2024-04-22T01:13:08.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200106a0 con 0x7fe928107460 2024-04-22T01:13:08.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.846+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92819ff40 con 0x7fe928107460 2024-04-22T01:13:08.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.847+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9280fd6e0 con 0x7fe928107460 2024-04-22T01:13:08.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.848+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fe920004b40 con 0x7fe928107460 2024-04-22T01:13:08.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.849+0000 7fe92c8c4640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9100a2c20 0x7fe9100a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:08.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.849+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe9200cfb10 con 0x7fe928107460 2024-04-22T01:13:08.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.849+0000 7fe926ffd640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9100a2c20 0x7fe9100a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:08.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.850+0000 7fe926ffd640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9100a2c20 0x7fe9100a50e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe91802c8e0 tx=0x7fe91802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:08.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:08.852+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe92009c500 con 0x7fe928107460 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: pgmap v379: 129 pgs: 129 active+clean; 652 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3088330195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3262273692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:08 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/700476028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:09.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.058+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe92802d3c0 con 0x7fe928107460 2024-04-22T01:13:09.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.059+0000 7fe92c8c4640 1 -- 172.21.15.38:0/1145403390 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe92009be00 con 0x7fe928107460 2024-04-22T01:13:09.058 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9100a2c20 msgr2=0x7fe9100a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9100a2c20 0x7fe9100a50e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe91802c8e0 tx=0x7fe91802df90 comp rx=0 tx=0).stop 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928107460 msgr2=0x7fe92819f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928107460 0x7fe92819f0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe92000da10 tx=0x7fe92000ded0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 shutdown_connections 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9100a2c20 0x7fe9100a50e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe928107460 0x7fe92819f0e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe928104b30 0x7fe928198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 --2- 172.21.15.38:0/1145403390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928102200 0x7fe928197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 >> 172.21.15.38:0/1145403390 conn(0x7fe9280fbbf0 msgr2=0x7fe928102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 shutdown_connections 2024-04-22T01:13:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.061+0000 7fe92d8c6640 1 -- 172.21.15.38:0/1145403390 wait complete. 2024-04-22T01:13:09.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: pgmap v379: 129 pgs: 129 active+clean; 652 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-22T01:13:09.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3088330195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:09.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3262273692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:09.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:08 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/700476028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:09.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: pgmap v379: 129 pgs: 129 active+clean; 652 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3088330195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3262273692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:09.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:08 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/700476028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:13:09.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 -- 172.21.15.38:0/262571613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 msgr2=0x7f102c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 --2- 172.21.15.38:0/262571613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c1098b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1024009ca0 tx=0x7f102402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 -- 172.21.15.38:0/262571613 shutdown_connections 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 --2- 172.21.15.38:0/262571613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c1098b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 --2- 172.21.15.38:0/262571613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f102c104b30 0x7f102c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 --2- 172.21.15.38:0/262571613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102c102200 0x7f102c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 -- 172.21.15.38:0/262571613 >> 172.21.15.38:0/262571613 conn(0x7f102c0fbbf0 msgr2=0x7f102c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 -- 172.21.15.38:0/262571613 shutdown_connections 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.183+0000 7f1033c9d640 1 -- 172.21.15.38:0/262571613 wait complete. 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 Processor -- start 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 -- start start 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102c102200 0x7f102c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f102c104b30 0x7f102c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d420 con 0x7f102c107460 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d2a0 con 0x7f102c104b30 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1033c9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d5a0 con 0x7f102c102200 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f103223a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f103223a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37400/0 (socket says 172.21.15.38:37400) 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f103223a640 1 -- 172.21.15.38:0/3664686533 learned_addr learned my addr 172.21.15.38:0/3664686533 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1031238640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f102c104b30 0x7f102c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.184+0000 7f1031a39640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102c102200 0x7f102c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f103223a640 1 -- 172.21.15.38:0/3664686533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102c102200 msgr2=0x7f102c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f103223a640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102c102200 0x7f102c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f103223a640 1 -- 172.21.15.38:0/3664686533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f102c104b30 msgr2=0x7f102c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f103223a640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f102c104b30 0x7f102c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f103223a640 1 -- 172.21.15.38:0/3664686533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102c19f6b0 con 0x7f102c107460 2024-04-22T01:13:09.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f103223a640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c19f040 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1024037040 tx=0x7f1024032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024041020 con 0x7f102c107460 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102c19f940 con 0x7f102c107460 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1024034cd0 con 0x7f102c107460 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024039040 con 0x7f102c107460 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.185+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102c19fe80 con 0x7f102c107460 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.186+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f102c0fd6e0 con 0x7f102c107460 2024-04-22T01:13:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.188+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f1024044020 con 0x7f102c107460 2024-04-22T01:13:09.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.189+0000 7f1022ffd640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f10180a2c20 0x7f10180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.189+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f10240f13f0 con 0x7f102c107460 2024-04-22T01:13:09.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.189+0000 7f1031a39640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f10180a2c20 0x7f10180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.189+0000 7f1031a39640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f10180a2c20 0x7f10180a50e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1028009bd0 tx=0x7f1028005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:09.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.196+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f102404b070 con 0x7f102c107460 2024-04-22T01:13:09.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.383+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f102c02d170 con 0x7f102c107460 2024-04-22T01:13:09.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.384+0000 7f1022ffd640 1 -- 172.21.15.38:0/3664686533 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f10240bdfc0 con 0x7f102c107460 2024-04-22T01:13:09.383 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:13:09.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f10180a2c20 msgr2=0x7f10180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f10180a2c20 0x7f10180a50e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1028009bd0 tx=0x7f1028005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:09.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 msgr2=0x7f102c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c19f040 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1024037040 tx=0x7f1024032f40 comp rx=0 tx=0).stop 2024-04-22T01:13:09.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 shutdown_connections 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f10180a2c20 0x7f10180a50e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f102c107460 0x7f102c19f040 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f102c104b30 0x7f102c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 --2- 172.21.15.38:0/3664686533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102c102200 0x7f102c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 >> 172.21.15.38:0/3664686533 conn(0x7f102c0fbbf0 msgr2=0x7f102c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 shutdown_connections 2024-04-22T01:13:09.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.386+0000 7f1033c9d640 1 -- 172.21.15.38:0/3664686533 wait complete. 2024-04-22T01:13:09.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:13:09.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 -- 172.21.15.38:0/1425776644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 msgr2=0x7fcdc4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/1425776644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc4108ac0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009ca0 tx=0x7fcdbc02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 -- 172.21.15.38:0/1425776644 shutdown_connections 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/1425776644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc4108ac0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/1425776644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 0x7fcdc4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/1425776644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 -- 172.21.15.38:0/1425776644 >> 172.21.15.38:0/1425776644 conn(0x7fcdc40fbbf0 msgr2=0x7fcdc40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 -- 172.21.15.38:0/1425776644 shutdown_connections 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 -- 172.21.15.38:0/1425776644 wait complete. 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 Processor -- start 2024-04-22T01:13:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.507+0000 7fcdca9b1640 1 -- start start 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdca9b1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc40694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdca9b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 0x7fcdc419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdca9b1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdca9b1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410c300 con 0x7fcdc4102230 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdca9b1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410c180 con 0x7fcdc4100680 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdca9b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410c480 con 0x7fcdc4101870 2024-04-22T01:13:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc8f4e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc40694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc8f4e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc419eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37412/0 (socket says 172.21.15.38:37412) 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc40694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43650/0 (socket says 172.21.15.38:43650) 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc8f4e640 1 -- 172.21.15.38:0/2054202423 learned_addr learned my addr 172.21.15.38:0/2054202423 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc37fe640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 0x7fcdc419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 -- 172.21.15.38:0/2054202423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 msgr2=0x7fcdc419c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 0x7fcdc419c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 -- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 msgr2=0x7fcdc419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc3fff640 1 -- 172.21.15.38:0/2054202423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc419f660 con 0x7fcdc4100680 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc8f4e640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:09.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.508+0000 7fcdc37fe640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 0x7fcdc419c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.509+0000 7fcdc3fff640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc40694a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcdc41016d0 tx=0x7fcdb400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.509+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4017070 con 0x7fcdc4100680 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.509+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdb4009c30 con 0x7fcdc4100680 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.509+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc419f950 con 0x7fcdc4100680 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.509+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb400f620 con 0x7fcdc4100680 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.509+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc419fea0 con 0x7fcdc4100680 2024-04-22T01:13:09.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.510+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc41079a0 con 0x7fcdc4100680 2024-04-22T01:13:09.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.511+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fcdb401a070 con 0x7fcdc4100680 2024-04-22T01:13:09.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.512+0000 7fcdc17fa640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcdac0a2c20 0x7fcdac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.512+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fcdb40cf180 con 0x7fcdc4100680 2024-04-22T01:13:09.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.512+0000 7fcdc37fe640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcdac0a2c20 0x7fcdac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.513+0000 7fcdc37fe640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcdac0a2c20 0x7fcdac0a50e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4069210 tx=0x7fcdb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:09.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.515+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcdb409bbb0 con 0x7fcdc4100680 2024-04-22T01:13:09.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.745+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdc402d170 con 0x7fcdc4100680 2024-04-22T01:13:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.746+0000 7fcdc17fa640 1 -- 172.21.15.38:0/2054202423 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7fcdb409b4b0 con 0x7fcdc4100680 2024-04-22T01:13:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:09.745 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:13:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:13:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcdac0a2c20 msgr2=0x7fcdac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcdac0a2c20 0x7fcdac0a50e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4069210 tx=0x7fcdb8005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 msgr2=0x7fcdc40694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc40694a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcdc41016d0 tx=0x7fcdb400dde0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 shutdown_connections 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcdac0a2c20 0x7fcdac0a50e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdc4102230 0x7fcdc419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdc4101870 0x7fcdc419c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 --2- 172.21.15.38:0/2054202423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4100680 0x7fcdc40694a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 >> 172.21.15.38:0/2054202423 conn(0x7fcdc40fbbf0 msgr2=0x7fcdc4104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 shutdown_connections 2024-04-22T01:13:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.748+0000 7fcdca9b1640 1 -- 172.21.15.38:0/2054202423 wait complete. 2024-04-22T01:13:09.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:13:09.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.864+0000 7f1e7a43b640 1 -- 172.21.15.38:0/1576168948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 msgr2=0x7f1e741098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.864+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/1576168948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e741098b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009ca0 tx=0x7f1e6c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 -- 172.21.15.38:0/1576168948 shutdown_connections 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/1576168948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e741098b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/1576168948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e74104b30 0x7f1e74106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/1576168948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e74102200 0x7f1e741045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 -- 172.21.15.38:0/1576168948 >> 172.21.15.38:0/1576168948 conn(0x7f1e740fbbf0 msgr2=0x7f1e740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 -- 172.21.15.38:0/1576168948 shutdown_connections 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 -- 172.21.15.38:0/1576168948 wait complete. 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 Processor -- start 2024-04-22T01:13:09.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 -- start start 2024-04-22T01:13:09.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.865+0000 7f1e7a43b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e74102200 0x7f1e74197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e7a43b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e74104b30 0x7f1e74197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e7a43b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e7419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e7a43b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410d420 con 0x7f1e74107460 2024-04-22T01:13:09.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e7a43b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410d2a0 con 0x7f1e74104b30 2024-04-22T01:13:09.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e7a43b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410d5a0 con 0x7f1e74102200 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e7419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e74104b30 0x7f1e74197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e7419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37432/0 (socket says 172.21.15.38:37432) 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 -- 172.21.15.38:0/2988082507 learned_addr learned my addr 172.21.15.38:0/2988082507 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e73fff640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e74102200 0x7f1e74197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 -- 172.21.15.38:0/2988082507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e74102200 msgr2=0x7f1e74197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e74102200 0x7f1e74197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 -- 172.21.15.38:0/2988082507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e74104b30 msgr2=0x7f1e74197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e74104b30 0x7f1e74197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 -- 172.21.15.38:0/2988082507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7419f690 con 0x7f1e74107460 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e789d8640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e7419f020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c037040 tx=0x7f1e6c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.866+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c041020 con 0x7f1e74107460 2024-04-22T01:13:09.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.867+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6c034d90 con 0x7f1e74107460 2024-04-22T01:13:09.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.867+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7419f920 con 0x7f1e74107460 2024-04-22T01:13:09.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.867+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c039040 con 0x7f1e74107460 2024-04-22T01:13:09.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.867+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7419fe10 con 0x7f1e74107460 2024-04-22T01:13:09.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.868+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e740fd6e0 con 0x7f1e74107460 2024-04-22T01:13:09.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.869+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f1e6c044020 con 0x7f1e74107460 2024-04-22T01:13:09.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.870+0000 7f1e717fa640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e600a2c20 0x7f1e600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:09.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.870+0000 7f1e73fff640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e600a2c20 0x7f1e600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:09.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.870+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f1e6c0f18a0 con 0x7f1e74107460 2024-04-22T01:13:09.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.870+0000 7f1e73fff640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e600a2c20 0x7f1e600a50e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009bd0 tx=0x7f1e64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:09.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:09.874+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e6c04b020 con 0x7f1e74107460 2024-04-22T01:13:10.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.067+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e7402d2f0 con 0x7f1e74107460 2024-04-22T01:13:10.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.068+0000 7f1e717fa640 1 -- 172.21.15.38:0/2988082507 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7f1e6c0be500 con 0x7f1e74107460 2024-04-22T01:13:10.068 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:10.068 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:13:10.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e600a2c20 msgr2=0x7f1e600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e600a2c20 0x7f1e600a50e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009bd0 tx=0x7f1e64005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 msgr2=0x7f1e7419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e7419f020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c037040 tx=0x7f1e6c032f20 comp rx=0 tx=0).stop 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 shutdown_connections 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1e600a2c20 0x7f1e600a50e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e74107460 0x7f1e7419f020 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e74104b30 0x7f1e74197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 --2- 172.21.15.38:0/2988082507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e74102200 0x7f1e74197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 >> 172.21.15.38:0/2988082507 conn(0x7f1e740fbbf0 msgr2=0x7f1e74102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 shutdown_connections 2024-04-22T01:13:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.070+0000 7f1e7a43b640 1 -- 172.21.15.38:0/2988082507 wait complete. 2024-04-22T01:13:10.081 DEBUG:tasks.fwd_scrub:Yielding 2024-04-22T01:13:10.082 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-22T01:13:10.092 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:13:10.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-22T01:13:10.092 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-22T01:13:10.093 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T01:13:10.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:10.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T01:13:10.095 INFO:teuthology.run_tasks:Running task workunit... 2024-04-22T01:13:10.107 INFO:tasks.workunit:Pulling workunits from ref 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T01:13:10.107 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T01:13:10.107 INFO:tasks.workunit:timeout=3h 2024-04-22T01:13:10.107 INFO:tasks.workunit:cleanup=True 2024-04-22T01:13:10.107 DEBUG:teuthology.orchestra.run.smithi038:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T01:13:10.124 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-22T01:13:10.124 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:13:10.124 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T01:13:10.124 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 01:11:01.014047011 +0000 2024-04-22T01:13:10.125 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 01:11:01.014047011 +0000 2024-04-22T01:13:10.125 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 01:13:01.481341866 +0000 2024-04-22T01:13:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 01:11:01.014047011 +0000 2024-04-22T01:13:10.125 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T01:13:10.125 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T01:13:10.197 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T01:13:10.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 -- 172.21.15.38:0/119270375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec102980 msgr2=0x7fd3ec102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/119270375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec102980 0x7fd3ec102e00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009900 tx=0x7fd3e802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 -- 172.21.15.38:0/119270375 shutdown_connections 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/119270375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ec103340 0x7fd3ec109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/119270375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec102980 0x7fd3ec102e00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/119270375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ec101790 0x7fd3ec101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 -- 172.21.15.38:0/119270375 >> 172.21.15.38:0/119270375 conn(0x7fd3ec0fcd20 msgr2=0x7fd3ec0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 -- 172.21.15.38:0/119270375 shutdown_connections 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.203+0000 7fd3f3d35640 1 -- 172.21.15.38:0/119270375 wait complete. 2024-04-22T01:13:10.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 Processor -- start 2024-04-22T01:13:10.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 -- start start 2024-04-22T01:13:10.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ec101790 0x7fd3ec197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ec102980 0x7fd3ec197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 0x7fd3ec19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec10d410 con 0x7fd3ec103340 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec10d290 con 0x7fd3ec101790 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f3d35640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec10d590 con 0x7fd3ec102980 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 0x7fd3ec19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 0x7fd3ec19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37448/0 (socket says 172.21.15.38:37448) 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 -- 172.21.15.38:0/3332299611 learned_addr learned my addr 172.21.15.38:0/3332299611 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ec101790 0x7fd3ec197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ec102980 0x7fd3ec197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 -- 172.21.15.38:0/3332299611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ec102980 msgr2=0x7fd3ec197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ec102980 0x7fd3ec197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 -- 172.21.15.38:0/3332299611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ec101790 msgr2=0x7fd3ec197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ec101790 0x7fd3ec197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.204+0000 7fd3f22d2640 1 -- 172.21.15.38:0/3332299611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ec19f650 con 0x7fd3ec103340 2024-04-22T01:13:10.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.205+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 0x7fd3ec19efe0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400cce0 tx=0x7fd3e400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.205+0000 7fd3e2ffd640 1 -- 172.21.15.38:0/3332299611 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e400f450 con 0x7fd3ec103340 2024-04-22T01:13:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.205+0000 7fd3e2ffd640 1 -- 172.21.15.38:0/3332299611 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e400fad0 con 0x7fd3ec103340 2024-04-22T01:13:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.205+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ec19f940 con 0x7fd3ec103340 2024-04-22T01:13:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.205+0000 7fd3e2ffd640 1 -- 172.21.15.38:0/3332299611 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4018c10 con 0x7fd3ec103340 2024-04-22T01:13:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.205+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ec19fe10 con 0x7fd3ec103340 2024-04-22T01:13:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.207+0000 7fd3e2ffd640 1 -- 172.21.15.38:0/3332299611 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fd3e400f5f0 con 0x7fd3ec103340 2024-04-22T01:13:10.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.208+0000 7fd3e2ffd640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3d80a2c20 0x7fd3d80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.208+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3d80a2c20 0x7fd3d80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.208+0000 7fd3e2ffd640 1 -- 172.21.15.38:0/3332299611 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd3e40cfbe0 con 0x7fd3ec103340 2024-04-22T01:13:10.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.208+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3d80a2c20 0x7fd3d80a50e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0058c0 tx=0x7fd3dc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.208+0000 7fd3f69f4740 1 Processor -- start 2024-04-22T01:13:10.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 -- start start 2024-04-22T01:13:10.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f651dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f656e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f659990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56142f5b6170 con 0x56142f6519a0 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56142f55e7c0 con 0x56142f657500 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56142f5b2130 con 0x56142f652390 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f651dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f651dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37454/0 (socket says 172.21.15.38:37454) 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3673450920 learned_addr learned my addr 172.21.15.38:0/3673450920 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f659990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f656e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3673450920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 msgr2=0x56142f656e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.209+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f656e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3673450920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 msgr2=0x56142f659990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f659990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3673450920 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56142f56fe60 con 0x56142f6519a0 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f651dc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ff00 tx=0x7fd3dc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45531c3191ae7b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3d7fff640 1 -- 172.21.15.38:0/3673450920 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc016070 con 0x56142f6519a0 2024-04-22T01:13:10.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3d7fff640 1 -- 172.21.15.38:0/3673450920 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc01b900 con 0x56142f6519a0 2024-04-22T01:13:10.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.210+0000 7fd3d7fff640 1 -- 172.21.15.38:0/3673450920 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc01c8c0 con 0x56142f6519a0 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3673450920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 msgr2=0x56142f651dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f651dc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ff00 tx=0x7fd3dc012000 comp rx=0 tx=0).stop 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3673450920 shutdown_connections 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f659990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f656e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3673450920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f651dc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3673450920 >> 172.21.15.38:0/3673450920 conn(0x56142f6194f0 msgr2=0x56142f64d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3673450920 shutdown_connections 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3673450920 wait complete. 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 -- start start 2024-04-22T01:13:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.211+0000 7fd3f69f4740 1 Processor -- start 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f65be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f65c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f663400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56142f65a4b0 con 0x56142f6519a0 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56142f571d40 con 0x56142f657500 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56142f65a630 con 0x56142f652390 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f69f4740 10 client.0 _ll_register_callbacks cb 0x56142f5ad820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f65be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f65be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37464/0 (socket says 172.21.15.38:37464) 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3618518486 learned_addr learned my addr 172.21.15.38:0/3618518486 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f65c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f663400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3618518486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 msgr2=0x56142f65c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f65c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 msgr2=0x56142f663400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f663400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/3618518486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56142f3927e0 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f65be80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00a8e0 tx=0x7fd3dc01b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a57ec6d9b94624 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.212+0000 7fd3d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3d57fa640 1 -- 172.21.15.38:0/3618518486 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc033060 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3d57fa640 1 -- 172.21.15.38:0/3618518486 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd3dc01bdf0 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3dc01b900 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3d57fa640 1 -- 172.21.15.38:0/3618518486 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3dc03a050 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 10 client.15654 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56142f3f2f70 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3d57fa640 1 -- 172.21.15.38:0/3618518486 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc01b900 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3d57fa640 1 -- 172.21.15.38:0/3618518486 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3dc02d030 con 0x56142f6519a0 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 10 client.15654 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 10 client.15654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 20 client.15654 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f667610 0x56142f669a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 4 client.15654 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.213+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56142f58a450 con 0x56142f667610 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.214+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f667610 0x56142f669a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.214+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f667610 0x56142f669a00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b689f34ddb6406fa server_cookie=5c4d73c3d0a227ef in_seq=0 out_seq=0 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.214+0000 7fd3d57fa640 10 client.15654 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3d57fa640 1 -- 172.21.15.38:0/3618518486 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56142f58a450 con 0x56142f667610 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3d57fa640 10 client.15654 handle_command_reply: tid=0 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 client.15654 shutdown 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 msgr2=0x56142f65be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f65be80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00a8e0 tx=0x7fd3dc01b740 comp rx=0 tx=0).stop 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 shutdown_connections 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f667610 0x56142f669a00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f663400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f65c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3d57fa640 0 client.15654 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/3618518486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f65be80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 >> 172.21.15.38:0/3618518486 conn(0x56142f6194f0 msgr2=0x56142f64c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 shutdown_connections 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 1 -- 172.21.15.38:0/3618518486 wait complete. 2024-04-22T01:13:10.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.215+0000 7fd3f69f4740 20 client.15654 trim_cache size 0 max 16384 2024-04-22T01:13:10.219 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T01:13:10.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 Processor -- start 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 -- start start 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f650de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f6633c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f65c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1aa3f0 con 0x56142f6519a0 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1abf30 con 0x56142f657500 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1aaac0 con 0x56142f652390 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.233+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f650de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f650de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37472/0 (socket says 172.21.15.38:37472) 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/1989988513 learned_addr learned my addr 172.21.15.38:0/1989988513 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f65c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f6633c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/1989988513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 msgr2=0x56142f6633c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f6633c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/1989988513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 msgr2=0x56142f65c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f65c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/1989988513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56142f28a170 con 0x56142f6519a0 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f6633c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f650de0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc010010 tx=0x7fd3dc00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccac116036a4882e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3d77fe640 1 -- 172.21.15.38:0/1989988513 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc033070 con 0x56142f6519a0 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3d77fe640 1 -- 172.21.15.38:0/1989988513 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc01b7b0 con 0x56142f6519a0 2024-04-22T01:13:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.234+0000 7fd3d77fe640 1 -- 172.21.15.38:0/1989988513 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc016040 con 0x56142f6519a0 2024-04-22T01:13:10.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 -- 172.21.15.38:0/1989988513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 msgr2=0x56142f650de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f650de0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc010010 tx=0x7fd3dc00fb30 comp rx=0 tx=0).stop 2024-04-22T01:13:10.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 -- 172.21.15.38:0/1989988513 shutdown_connections 2024-04-22T01:13:10.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f657500 0x56142f65c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f652390 0x56142f6633c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/1989988513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f650de0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 -- 172.21.15.38:0/1989988513 >> 172.21.15.38:0/1989988513 conn(0x56142f6194f0 msgr2=0x56142f65ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 -- 172.21.15.38:0/1989988513 shutdown_connections 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.235+0000 7fd3f69f4740 1 -- 172.21.15.38:0/1989988513 wait complete. 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 -- start start 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 Processor -- start 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f661170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f652390 0x56142f6616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f657500 0x56142f6626c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1ab860 con 0x56142f6519a0 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1ab190 con 0x56142f652390 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1a83b0 con 0x56142f657500 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f69f4740 10 client.0 _ll_register_callbacks cb 0x7fd3e4018400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f661170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f1ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f661170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37476/0 (socket says 172.21.15.38:37476) 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/2286606061 learned_addr learned my addr 172.21.15.38:0/2286606061 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f652390 0x56142f6616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.236+0000 7fd3f22d2640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f657500 0x56142f6626c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/2286606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f657500 msgr2=0x56142f6626c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f657500 0x56142f6626c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f652390 msgr2=0x56142f6616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f652390 0x56142f6616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f1ad1640 1 -- 172.21.15.38:0/2286606061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56142f662cb0 con 0x56142f6519a0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f1ad1640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f661170 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc010ec0 tx=0x7fd3dc010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c329b892c83e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3d57fa640 1 -- 172.21.15.38:0/2286606061 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc016070 con 0x56142f6519a0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3d57fa640 1 -- 172.21.15.38:0/2286606061 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd3dc004a90 con 0x56142f6519a0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3dc01b7b0 con 0x56142f6519a0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3d57fa640 1 -- 172.21.15.38:0/2286606061 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3dc052050 con 0x56142f6519a0 2024-04-22T01:13:10.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f69f4740 10 client.15660 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56142f6ce160 con 0x56142f6519a0 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.237+0000 7fd3d57fa640 1 -- 172.21.15.38:0/2286606061 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc01b7b0 con 0x56142f6519a0 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3d57fa640 1 -- 172.21.15.38:0/2286606061 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3dc02d040 con 0x56142f6519a0 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f69f4740 10 client.15660 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f69f4740 10 client.15660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f69f4740 20 client.15660 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f6cea40 0x56142f6d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f69f4740 4 client.15660 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56142f5c36b0 con 0x56142f6cea40 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f6cea40 0x56142f6d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3d57fa640 10 client.15660 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.238+0000 7fd3f12d0640 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f6cea40 0x56142f6d0e60 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78719ebff515e74a server_cookie=8fb8b50690090658 in_seq=0 out_seq=0 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.239+0000 7fd3d57fa640 1 -- 172.21.15.38:0/2286606061 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56142f5c36b0 con 0x56142f6cea40 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.239+0000 7fd3d57fa640 10 client.15660 handle_command_reply: tid=0 2024-04-22T01:13:10.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.239+0000 7fd3f69f4740 1 client.15660 shutdown 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 msgr2=0x56142f661170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f661170 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc010ec0 tx=0x7fd3dc010ef0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 shutdown_connections 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56142f6cea40 0x56142f6d0e60 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56142f657500 0x56142f6626c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56142f652390 0x56142f6616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3d57fa640 0 client.15660 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 --2- 172.21.15.38:0/2286606061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56142f6519a0 0x56142f661170 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 >> 172.21.15.38:0/2286606061 conn(0x56142f6194f0 msgr2=0x56142f66e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 shutdown_connections 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 1 -- 172.21.15.38:0/2286606061 wait complete. 2024-04-22T01:13:10.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.240+0000 7fd3f69f4740 20 client.15660 trim_cache size 0 max 16384 2024-04-22T01:13:10.244 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:13:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T01:13:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "f326b629-0e37-4a11-aacc-40f02b586dca", 2024-04-22T01:13:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T01:13:10.244 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:13:10.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3d80a2c20 msgr2=0x7fd3d80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3d80a2c20 0x7fd3d80a50e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0058c0 tx=0x7fd3dc00a2d0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 msgr2=0x7fd3ec19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 0x7fd3ec19efe0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400cce0 tx=0x7fd3e400a620 comp rx=0 tx=0).stop 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 shutdown_connections 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3d80a2c20 0x7fd3d80a50e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ec103340 0x7fd3ec19efe0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ec102980 0x7fd3ec197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 --2- 172.21.15.38:0/3332299611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ec101790 0x7fd3ec197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 >> 172.21.15.38:0/3332299611 conn(0x7fd3ec0fcd20 msgr2=0x7fd3ec105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 shutdown_connections 2024-04-22T01:13:10.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.241+0000 7fd3f3d35640 1 -- 172.21.15.38:0/3332299611 wait complete. 2024-04-22T01:13:10.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:13:10.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:13:10.314 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1145403390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3664686533' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:10 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2054202423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:10.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1145403390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:10.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3664686533' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:10.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:13:10.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:13:10.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:10 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2054202423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:10.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 -- 172.21.15.38:0/1854856209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0102200 msgr2=0x7f15c01045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/1854856209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0102200 0x7f15c01045f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009900 tx=0x7f15b002bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:10.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 -- 172.21.15.38:0/1854856209 shutdown_connections 2024-04-22T01:13:10.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/1854856209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0107460 0x7f15c01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/1854856209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15c0104b30 0x7f15c0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/1854856209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0102200 0x7f15c01045f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 -- 172.21.15.38:0/1854856209 >> 172.21.15.38:0/1854856209 conn(0x7f15c00fbbf0 msgr2=0x7f15c00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 -- 172.21.15.38:0/1854856209 shutdown_connections 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.410+0000 7f15c5b1e640 1 -- 172.21.15.38:0/1854856209 wait complete. 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 Processor -- start 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 -- start start 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0102200 0x7f15c0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 0x7f15c0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15c0107460 0x7f15c019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15c010d400 con 0x7f15c0104b30 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15c010d280 con 0x7f15c0102200 2024-04-22T01:13:10.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15c5b1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15c010d580 con 0x7f15c0107460 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 0x7f15c0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 0x7f15c0198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37490/0 (socket says 172.21.15.38:37490) 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 -- 172.21.15.38:0/3880498331 learned_addr learned my addr 172.21.15.38:0/3880498331 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15bffff640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15c0107460 0x7f15c019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 -- 172.21.15.38:0/3880498331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15c0107460 msgr2=0x7f15c019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15c0107460 0x7f15c019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 -- 172.21.15.38:0/3880498331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0102200 msgr2=0x7f15c0197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0102200 0x7f15c0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0102200 0x7f15c0197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15beffd640 1 -- 172.21.15.38:0/3880498331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c019f730 con 0x7f15c0104b30 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.411+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0102200 0x7f15c0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.412+0000 7f15beffd640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 0x7f15c0198050 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f15b400d770 tx=0x7f15b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.412+0000 7f15bcff9640 1 -- 172.21.15.38:0/3880498331 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4017070 con 0x7f15c0104b30 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.412+0000 7f15bcff9640 1 -- 172.21.15.38:0/3880498331 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b4009c30 con 0x7f15c0104b30 2024-04-22T01:13:10.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.412+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c019fa20 con 0x7f15c0104b30 2024-04-22T01:13:10.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.412+0000 7f15bcff9640 1 -- 172.21.15.38:0/3880498331 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b400f620 con 0x7f15c0104b30 2024-04-22T01:13:10.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.412+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c019ff70 con 0x7f15c0104b30 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15bcff9640 1 -- 172.21.15.38:0/3880498331 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f15b401a070 con 0x7f15c0104b30 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 Processor -- start 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 -- start start 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6c6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e10c62b560 con 0x55e10c6cbab0 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e10c5d3730 con 0x55e10c6c6940 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15c87e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e10c627520 con 0x55e10c6c5f50 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37504/0 (socket says 172.21.15.38:37504) 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15beffd640 1 -- 172.21.15.38:0/2235719891 learned_addr learned my addr 172.21.15.38:0/2235719891 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15bcff9640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15a80a2c20 0x7f15a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15bffff640 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.415+0000 7f15beffd640 1 -- 172.21.15.38:0/2235719891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 msgr2=0x55e10c6c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15beffd640 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15beffd640 1 -- 172.21.15.38:0/2235719891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 msgr2=0x55e10c6cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15beffd640 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15bcff9640 1 -- 172.21.15.38:0/3880498331 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f15b40cf160 con 0x7f15c0104b30 2024-04-22T01:13:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15beffd640 1 -- 172.21.15.38:0/2235719891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10c4097d0 con 0x55e10c6cbab0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15bffff640 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15bffff640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15a80a2c20 0x7f15a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15beffd640 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6cdf40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f15b40978f0 tx=0x7f15b4010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf87597c4008e5f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15a5ffb640 1 -- 172.21.15.38:0/2235719891 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4021070 con 0x55e10c6cbab0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15a5ffb640 1 -- 172.21.15.38:0/2235719891 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b4009de0 con 0x55e10c6cbab0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.416+0000 7f15a5ffb640 1 -- 172.21.15.38:0/2235719891 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b409f840 con 0x55e10c6cbab0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- 172.21.15.38:0/2235719891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 msgr2=0x55e10c6cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15bffff640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15a80a2c20 0x7f15a80a50e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x55e10c6c71a0 tx=0x7f15b8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6cdf40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f15b40978f0 tx=0x7f15b4010cd0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- 172.21.15.38:0/2235719891 shutdown_connections 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6cdf40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2235719891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- 172.21.15.38:0/2235719891 >> 172.21.15.38:0/2235719891 conn(0x55e10c6c14c0 msgr2=0x55e10c6c18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- 172.21.15.38:0/2235719891 shutdown_connections 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- 172.21.15.38:0/2235719891 wait complete. 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- start start 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 Processor -- start 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c6940 0x55e10c6d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6d78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e10c6ce9a0 con 0x55e10c6cbab0 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e10c5e6e00 con 0x55e10c6c5f50 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.417+0000 7f15c87e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e10c6ceb20 con 0x55e10c6c6940 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15c87e8740 10 client.0 _ll_register_callbacks cb 0x55e10c621eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6d78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6d78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37516/0 (socket says 172.21.15.38:37516) 2024-04-22T01:13:10.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 -- 172.21.15.38:0/2668935059 learned_addr learned my addr 172.21.15.38:0/2668935059 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15beffd640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c6940 0x55e10c6d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 -- 172.21.15.38:0/2668935059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c6940 msgr2=0x55e10c6d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c6940 0x55e10c6d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 -- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 msgr2=0x55e10c6d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 -- 172.21.15.38:0/2668935059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10c33f3f0 con 0x55e10c6cbab0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15bffff640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6d78c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f15b8013950 tx=0x7f15b8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21ede31a8a2424b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f159effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f159effd640 1 -- 172.21.15.38:0/2668935059 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b801ed70 con 0x55e10c6cbab0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f159effd640 1 -- 172.21.15.38:0/2668935059 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f15b8028470 con 0x55e10c6cbab0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15b4009de0 con 0x55e10c6cbab0 2024-04-22T01:13:10.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.418+0000 7f159effd640 1 -- 172.21.15.38:0/2668935059 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f15b8028670 con 0x55e10c6cbab0 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 10 client.15672 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e10c2ff0f0 con 0x55e10c6cbab0 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f159effd640 1 -- 172.21.15.38:0/2668935059 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b4009de0 con 0x55e10c6cbab0 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f159effd640 1 -- 172.21.15.38:0/2668935059 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f15b800f040 con 0x55e10c6cbab0 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 10 client.15672 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 10 client.15672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 20 client.15672 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c6dc210 0x55e10c6de680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 4 client.15672 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e10c638aa0 con 0x55e10c6dc210 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.419+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c6dc210 0x55e10c6de680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.420+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c6dc210 0x55e10c6de680 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53aa926b783cb1fe server_cookie=c3f2f16275c28775 in_seq=0 out_seq=0 2024-04-22T01:13:10.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.420+0000 7f159effd640 10 client.15672 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f159effd640 1 -- 172.21.15.38:0/2668935059 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e10c638aa0 con 0x55e10c6dc210 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f159effd640 10 client.15672 handle_command_reply: tid=0 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 client.15672 shutdown 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 msgr2=0x55e10c6d78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6d78c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f15b8013950 tx=0x7f15b8004ab0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 shutdown_connections 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c6dc210 0x55e10c6de680 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6d78c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c6940 0x55e10c6d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2668935059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f159effd640 0 client.15672 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 >> 172.21.15.38:0/2668935059 conn(0x55e10c6c14c0 msgr2=0x55e10c6c06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 shutdown_connections 2024-04-22T01:13:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 1 -- 172.21.15.38:0/2668935059 wait complete. 2024-04-22T01:13:10.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.421+0000 7f15c87e8740 20 client.15672 trim_cache size 0 max 16384 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.438+0000 7f15c87e8740 1 Processor -- start 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.438+0000 7f15c87e8740 1 -- start start 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6bc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15c87e8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01aa660 con 0x55e10c6cbab0 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15c87e8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01ac1a0 con 0x55e10c6c6940 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15c87e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01aad30 con 0x55e10c6c5f50 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6bfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37518/0 (socket says 172.21.15.38:37518) 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 -- 172.21.15.38:0/3473797801 learned_addr learned my addr 172.21.15.38:0/3473797801 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6bc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15beffd640 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 -- 172.21.15.38:0/3473797801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 msgr2=0x55e10c6bc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6bc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 -- 172.21.15.38:0/3473797801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 msgr2=0x55e10c6d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6d77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 -- 172.21.15.38:0/3473797801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10c6c12f0 con 0x55e10c6cbab0 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15bffff640 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6bfe50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f15b801fc50 tx=0x7f15b8005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5aafc26427b57c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15a57fa640 1 -- 172.21.15.38:0/3473797801 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b801e7b0 con 0x55e10c6cbab0 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15a57fa640 1 -- 172.21.15.38:0/3473797801 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b8023070 con 0x55e10c6cbab0 2024-04-22T01:13:10.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.439+0000 7f15a57fa640 1 -- 172.21.15.38:0/3473797801 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b8028cb0 con 0x55e10c6cbab0 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 -- 172.21.15.38:0/3473797801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 msgr2=0x55e10c6bfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6bfe50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f15b801fc50 tx=0x7f15b8005be0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 -- 172.21.15.38:0/3473797801 shutdown_connections 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6cbab0 0x55e10c6bfe50 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c6940 0x55e10c6d77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 --2- 172.21.15.38:0/3473797801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6c5f50 0x55e10c6bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 -- 172.21.15.38:0/3473797801 >> 172.21.15.38:0/3473797801 conn(0x55e10c6c14c0 msgr2=0x55e10c6c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 -- 172.21.15.38:0/3473797801 shutdown_connections 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 -- 172.21.15.38:0/3473797801 wait complete. 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.440+0000 7f15c87e8740 1 -- start start 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 Processor -- start 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 0x55e10c6d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6cbab0 0x55e10c6d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01abad0 con 0x55e10c6c6940 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01ab400 con 0x55e10c6c5f50 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15c01a8620 con 0x55e10c6cbab0 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 10 client.0 _ll_register_callbacks cb 0x7f15b8009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 0x55e10c6d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 0x55e10c6d55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37528/0 (socket says 172.21.15.38:37528) 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 -- 172.21.15.38:0/2429600691 learned_addr learned my addr 172.21.15.38:0/2429600691 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15bffff640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6cbab0 0x55e10c6d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 -- 172.21.15.38:0/2429600691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6cbab0 msgr2=0x55e10c6d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6cbab0 0x55e10c6d64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 -- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 msgr2=0x55e10c6d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 -- 172.21.15.38:0/2429600691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10c6dbff0 con 0x55e10c6c6940 2024-04-22T01:13:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15beffd640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 0x55e10c6d55b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f15b40cccc0 tx=0x7f15b40cc6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5dae951d6afd38e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f159effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f159effd640 1 -- 172.21.15.38:0/2429600691 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4021040 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f159effd640 1 -- 172.21.15.38:0/2429600691 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f15b40cc990 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.441+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15b8023070 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f159effd640 1 -- 172.21.15.38:0/2429600691 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f15b4018260 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 10 client.15678 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e10c7427c0 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f159effd640 1 -- 172.21.15.38:0/2429600691 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b8023070 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f159effd640 1 -- 172.21.15.38:0/2429600691 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f15b4097b00 con 0x55e10c6c6940 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 10 client.15678 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 10 client.15678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 20 client.15678 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c742cd0 0x55e10c7450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 4 client.15678 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e10c632df0 con 0x55e10c742cd0 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.442+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c742cd0 0x55e10c7450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f159effd640 10 client.15678 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15bf7fe640 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c742cd0 0x55e10c7450c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66376a45e7683b54 server_cookie=882c0db567c22e30 in_seq=0 out_seq=0 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f159effd640 1 -- 172.21.15.38:0/2429600691 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e10c632df0 con 0x55e10c742cd0 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f159effd640 10 client.15678 handle_command_reply: tid=0 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 client.15678 shutdown 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 msgr2=0x55e10c6d55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 0x55e10c6d55b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f15b40cccc0 tx=0x7f15b40cc6f0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 shutdown_connections 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e10c742cd0 0x55e10c7450c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e10c6cbab0 0x55e10c6d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e10c6c6940 0x55e10c6d55b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f159effd640 0 client.15678 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 --2- 172.21.15.38:0/2429600691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e10c6c5f50 0x55e10c6d5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 >> 172.21.15.38:0/2429600691 conn(0x55e10c6c14c0 msgr2=0x55e10c6c4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 shutdown_connections 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 1 -- 172.21.15.38:0/2429600691 wait complete. 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.443+0000 7f15c87e8740 20 client.15678 trim_cache size 0 max 16384 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stdout: "f326b629-0e37-4a11-aacc-40f02b586dca": { 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T01:13:10.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "f326b629-0e37-4a11-aacc-40f02b586dca", 2024-04-22T01:13:10.450 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T01:13:10.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:13:10.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:13:10.450 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:13:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15a80a2c20 msgr2=0x7f15a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15a80a2c20 0x7f15a80a50e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x55e10c6c71a0 tx=0x7f15b8004a40 comp rx=0 tx=0).stop 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 msgr2=0x7f15c0198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 0x7f15c0198050 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f15b400d770 tx=0x7f15b400dc30 comp rx=0 tx=0).stop 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 shutdown_connections 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15a80a2c20 0x7f15a80a50e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15c0107460 0x7f15c019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15c0104b30 0x7f15c0198050 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 --2- 172.21.15.38:0/3880498331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15c0102200 0x7f15c0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 >> 172.21.15.38:0/3880498331 conn(0x7f15c00fbbf0 msgr2=0x7f15c0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 shutdown_connections 2024-04-22T01:13:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:10.444+0000 7f15c5b1e640 1 -- 172.21.15.38:0/3880498331 wait complete. 2024-04-22T01:13:10.454 INFO:tasks.cephfs.filesystem:scrub status for tag:f326b629-0e37-4a11-aacc-40f02b586dca - {'path': '/', 'tag': 'f326b629-0e37-4a11-aacc-40f02b586dca', 'options': 'recursive,force'} 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1145403390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3664686533' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:13:10.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:10 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2054202423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:11.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:11 smithi111 ceph-mon[31057]: pgmap v380: 129 pgs: 129 active+clean; 652 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T01:13:11.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:11 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2988082507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:11.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:11 smithi111 ceph-mon[31057]: scrub queued for path: / 2024-04-22T01:13:11.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:11 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:11.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:11 smithi111 ceph-mon[31057]: scrub summary: active paths [/] 2024-04-22T01:13:11.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:11 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:11 smithi038 ceph-mon[28043]: pgmap v380: 129 pgs: 129 active+clean; 652 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:11 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2988082507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:11 smithi038 ceph-mon[28043]: scrub queued for path: / 2024-04-22T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:11 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:11 smithi038 ceph-mon[28043]: scrub summary: active paths [/] 2024-04-22T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:11 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:11 smithi203 ceph-mon[31825]: pgmap v380: 129 pgs: 129 active+clean; 652 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T01:13:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:11 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2988082507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:13:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:11 smithi203 ceph-mon[31825]: scrub queued for path: / 2024-04-22T01:13:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:11 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:11 smithi203 ceph-mon[31825]: scrub summary: active paths [/] 2024-04-22T01:13:11.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:11 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:13.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:13 smithi111 ceph-mon[31057]: pgmap v381: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-04-22T01:13:13.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:13 smithi038 ceph-mon[28043]: pgmap v381: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-04-22T01:13:13.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:13 smithi203 ceph-mon[31825]: pgmap v381: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-04-22T01:13:15.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:15 smithi111 ceph-mon[31057]: pgmap v382: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-04-22T01:13:15.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:15 smithi038 ceph-mon[28043]: pgmap v382: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-04-22T01:13:15.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:15 smithi203 ceph-mon[31825]: pgmap v382: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-04-22T01:13:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:17 smithi111 ceph-mon[31057]: pgmap v383: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-22T01:13:17.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:17 smithi038 ceph-mon[28043]: pgmap v383: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-22T01:13:17.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:17 smithi203 ceph-mon[31825]: pgmap v383: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-22T01:13:19.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:19 smithi111 ceph-mon[31057]: pgmap v384: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-04-22T01:13:19.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:19 smithi038 ceph-mon[28043]: pgmap v384: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-04-22T01:13:19.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:19 smithi203 ceph-mon[31825]: pgmap v384: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-04-22T01:13:21.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:21 smithi111 ceph-mon[31057]: pgmap v385: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-22T01:13:21.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:21 smithi038 ceph-mon[28043]: pgmap v385: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-22T01:13:21.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:21 smithi203 ceph-mon[31825]: pgmap v385: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-22T01:13:22.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:22 smithi111 ceph-mon[31057]: scrub summary: idle 2024-04-22T01:13:22.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:22 smithi038 ceph-mon[28043]: scrub summary: idle 2024-04-22T01:13:22.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:22 smithi203 ceph-mon[31825]: scrub summary: idle 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 -- 172.21.15.38:0/28656306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04101890 msgr2=0x7fcf04101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 --2- 172.21.15.38:0/28656306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04101890 0x7fcf04101d10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fceec009900 tx=0x7fceec02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 -- 172.21.15.38:0/28656306 shutdown_connections 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 --2- 172.21.15.38:0/28656306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf04102250 0x7fcf04108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 --2- 172.21.15.38:0/28656306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04101890 0x7fcf04101d10 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 --2- 172.21.15.38:0/28656306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf041006a0 0x7fcf04100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 -- 172.21.15.38:0/28656306 >> 172.21.15.38:0/28656306 conn(0x7fcf040fbc10 msgr2=0x7fcf040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:23.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 -- 172.21.15.38:0/28656306 shutdown_connections 2024-04-22T01:13:23.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.237+0000 7fcf0907d640 1 -- 172.21.15.38:0/28656306 wait complete. 2024-04-22T01:13:23.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 Processor -- start 2024-04-22T01:13:23.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 -- start start 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf041006a0 0x7fcf041979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf04101890 0x7fcf04197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 0x7fcf0419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0410c320 con 0x7fcf04102250 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0410c1a0 con 0x7fcf041006a0 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0907d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0410c4a0 con 0x7fcf04101890 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 0x7fcf0419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 0x7fcf0419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53314/0 (socket says 172.21.15.38:53314) 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcf0359e640 1 -- 172.21.15.38:0/1769109159 learned_addr learned my addr 172.21.15.38:0/1769109159 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.238+0000 7fcefbfff640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf04101890 0x7fcf04197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf041006a0 0x7fcf041979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0359e640 1 -- 172.21.15.38:0/1769109159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf04101890 msgr2=0x7fcf04197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0359e640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf04101890 0x7fcf04197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0359e640 1 -- 172.21.15.38:0/1769109159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf041006a0 msgr2=0x7fcf041979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0359e640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf041006a0 0x7fcf041979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0359e640 1 -- 172.21.15.38:0/1769109159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0419f5d0 con 0x7fcf04102250 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0359e640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 0x7fcf0419ef60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00cce0 tx=0x7fcefc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf00d99640 1 -- 172.21.15.38:0/1769109159 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc00f450 con 0x7fcf04102250 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf00d99640 1 -- 172.21.15.38:0/1769109159 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcefc00fad0 con 0x7fcf04102250 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0419f8c0 con 0x7fcf04102250 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf00d99640 1 -- 172.21.15.38:0/1769109159 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc018c10 con 0x7fcf04102250 2024-04-22T01:13:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.239+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0419fd90 con 0x7fcf04102250 2024-04-22T01:13:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf00d99640 1 -- 172.21.15.38:0/1769109159 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fcefc00f5f0 con 0x7fcf04102250 2024-04-22T01:13:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 Processor -- start 2024-04-22T01:13:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 -- start start 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a6fd30 0x560462a70150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a75220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a77d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5604629d4b70 con 0x560462a70720 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56046297d470 con 0x560462a75890 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604629d0b30 con 0x560462a6fd30 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf02d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a75220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf02d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a75220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53326/0 (socket says 172.21.15.38:53326) 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcf02d9d640 1 -- 172.21.15.38:0/1089158537 learned_addr learned my addr 172.21.15.38:0/1089158537 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.242+0000 7fcefbfff640 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a6fd30 0x560462a70150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcefbfff640 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a77d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf00d99640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcef00a2cd0 0x7fcef00a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf02d9d640 1 -- 172.21.15.38:0/1089158537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a6fd30 msgr2=0x560462a70150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a6fd30 0x560462a70150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf02d9d640 1 -- 172.21.15.38:0/1089158537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 msgr2=0x560462a77d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a77d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf02d9d640 1 -- 172.21.15.38:0/1089158537 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604626e9210 con 0x560462a70720 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcefbfff640 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a77d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf0359e640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcef00a2cd0 0x7fcef00a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a75220 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009c20 tx=0x7fcef40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f05188f930d3849 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcf00d99640 1 -- 172.21.15.38:0/1769109159 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fcefc0cfc90 con 0x7fcf04102250 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcef97fa640 1 -- 172.21.15.38:0/1089158537 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4012450 con 0x560462a70720 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcef97fa640 1 -- 172.21.15.38:0/1089158537 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcef4012ad0 con 0x560462a70720 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.243+0000 7fcef97fa640 1 -- 172.21.15.38:0/1089158537 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef400fb80 con 0x560462a70720 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 -- 172.21.15.38:0/1089158537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 msgr2=0x560462a75220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0359e640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcef00a2cd0 0x7fcef00a5190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcefc09bcc0 tx=0x7fcefc0914a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a75220 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009c20 tx=0x7fcef40117b0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 -- 172.21.15.38:0/1089158537 shutdown_connections 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a77d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a75220 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/1089158537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a6fd30 0x560462a70150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 -- 172.21.15.38:0/1089158537 >> 172.21.15.38:0/1089158537 conn(0x560462a6b2a0 msgr2=0x560462a6b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 -- 172.21.15.38:0/1089158537 shutdown_connections 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 -- 172.21.15.38:0/1089158537 wait complete. 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 -- start start 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.244+0000 7fcf0bc69740 1 Processor -- start 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a7a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a70720 0x560462a7a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a815b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560462a78840 con 0x560462a6fd30 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560462990a10 con 0x560462a70720 2024-04-22T01:13:23.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560462a789c0 con 0x560462a75890 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf0bc69740 10 client.0 _ll_register_callbacks cb 0x5604629cbb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a7a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a7a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53334/0 (socket says 172.21.15.38:53334) 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 -- 172.21.15.38:0/3683584719 learned_addr learned my addr 172.21.15.38:0/3683584719 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a70720 0x560462a7a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a815b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 -- 172.21.15.38:0/3683584719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 msgr2=0x560462a815b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a815b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 -- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a70720 msgr2=0x560462a7a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a70720 0x560462a7a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 -- 172.21.15.38:0/3683584719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604627b5620 con 0x560462a6fd30 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fcefbfff640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a7a030 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fceec036040 tx=0x7fceec005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05d80f0f405acd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.245+0000 7fceea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fceea7fc640 1 -- 172.21.15.38:0/3683584719 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec048020 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fceea7fc640 1 -- 172.21.15.38:0/3683584719 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fceec005450 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef4012ad0 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fceea7fc640 1 -- 172.21.15.38:0/3683584719 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fceec053050 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 10 client.15690 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560462a84900 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fceea7fc640 1 -- 172.21.15.38:0/3683584719 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcef4012ad0 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fceea7fc640 1 -- 172.21.15.38:0/3683584719 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fceec003890 con 0x560462a6fd30 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 10 client.15690 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 20 client.15690 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462a85fd0 0x560462a88400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 4 client.15690 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.246+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604629e20b0 con 0x560462a85fd0 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.247+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462a85fd0 0x560462a88400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.247+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462a85fd0 0x560462a88400 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8b33125b61968e server_cookie=1cb35ef915737fb7 in_seq=0 out_seq=0 2024-04-22T01:13:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.247+0000 7fceea7fc640 10 client.15690 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fceea7fc640 1 -- 172.21.15.38:0/3683584719 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604629e20b0 con 0x560462a85fd0 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fceea7fc640 10 client.15690 handle_command_reply: tid=0 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 client.15690 shutdown 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 msgr2=0x560462a7a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a7a030 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fceec036040 tx=0x7fceec005d50 comp rx=0 tx=0).stop 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 shutdown_connections 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462a85fd0 0x560462a88400 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a815b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fceea7fc640 0 client.15690 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a70720 0x560462a7a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/3683584719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a7a030 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 >> 172.21.15.38:0/3683584719 conn(0x560462a6b2a0 msgr2=0x560462a6a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 shutdown_connections 2024-04-22T01:13:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 1 -- 172.21.15.38:0/3683584719 wait complete. 2024-04-22T01:13:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.248+0000 7fcf0bc69740 20 client.15690 trim_cache size 0 max 16384 2024-04-22T01:13:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 Processor -- start 2024-04-22T01:13:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 -- start start 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a795f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a70720 0x560462a7a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041aa330 con 0x560462a6fd30 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041abe70 con 0x560462a75890 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041aaa00 con 0x560462a70720 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.263+0000 7fcefbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a795f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53350/0 (socket says 172.21.15.38:53350) 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 -- 172.21.15.38:0/121483842 learned_addr learned my addr 172.21.15.38:0/121483842 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 -- 172.21.15.38:0/121483842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a70720 msgr2=0x560462a7a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a70720 0x560462a7a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a70720 0x560462a7a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 -- 172.21.15.38:0/121483842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 msgr2=0x560462a69c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 -- 172.21.15.38:0/121483842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560462a74cd0 con 0x560462a6fd30 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcefbfff640 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a795f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fceec001eb0 tx=0x7fceec0040a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bd902cab94304d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcef8ff9640 1 -- 172.21.15.38:0/121483842 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec048070 con 0x560462a6fd30 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcef8ff9640 1 -- 172.21.15.38:0/121483842 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceec005450 con 0x560462a6fd30 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.264+0000 7fcef8ff9640 1 -- 172.21.15.38:0/121483842 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec004990 con 0x560462a6fd30 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- 172.21.15.38:0/121483842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 msgr2=0x560462a795f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a795f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fceec001eb0 tx=0x7fceec0040a0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- 172.21.15.38:0/121483842 shutdown_connections 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a75890 0x560462a69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a70720 0x560462a7a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/121483842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a6fd30 0x560462a795f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- 172.21.15.38:0/121483842 >> 172.21.15.38:0/121483842 conn(0x560462a6b2a0 msgr2=0x560462a79c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- 172.21.15.38:0/121483842 shutdown_connections 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- 172.21.15.38:0/121483842 wait complete. 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- start start 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 Processor -- start 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a6fd30 0x560462a7e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a7e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a7f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041ab7a0 con 0x560462a70720 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041ab0d0 con 0x560462a6fd30 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.265+0000 7fcf0bc69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf041a82f0 con 0x560462a75890 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf0bc69740 10 client.0 _ll_register_callbacks cb 0x7fceec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a7e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a7e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53352/0 (socket says 172.21.15.38:53352) 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 -- 172.21.15.38:0/772641473 learned_addr learned my addr 172.21.15.38:0/772641473 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcefbfff640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a6fd30 0x560462a7e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcefbfff640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a7f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 -- 172.21.15.38:0/772641473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 msgr2=0x560462a7f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a7f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 -- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a6fd30 msgr2=0x560462a7e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a6fd30 0x560462a7e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 -- 172.21.15.38:0/772641473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560462a7fe60 con 0x560462a70720 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf02d9d640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a7e8d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009e50 tx=0x7fcef40096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cff7826fc5605da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fceea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fceea7fc640 1 -- 172.21.15.38:0/772641473 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4005330 con 0x560462a70720 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fceea7fc640 1 -- 172.21.15.38:0/772641473 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fcef40054d0 con 0x560462a70720 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceec005450 con 0x560462a70720 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fceea7fc640 1 -- 172.21.15.38:0/772641473 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcef40035f0 con 0x560462a70720 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf0bc69740 10 client.15696 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.266+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560462aec6d0 con 0x560462a70720 2024-04-22T01:13:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fceea7fc640 1 -- 172.21.15.38:0/772641473 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceec005450 con 0x560462a70720 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fceea7fc640 1 -- 172.21.15.38:0/772641473 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcef400f850 con 0x560462a70720 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcf0bc69740 10 client.15696 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcf0bc69740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcf0bc69740 20 client.15696 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462aecf70 0x560462aef390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcf0bc69740 4 client.15696 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604629dc400 con 0x560462aecf70 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcefbfff640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462aecf70 0x560462aef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fcefbfff640 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462aecf70 0x560462aef390 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9416629642ede673 server_cookie=d2413052a2959781 in_seq=0 out_seq=0 2024-04-22T01:13:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.267+0000 7fceea7fc640 10 client.15696 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.280+0000 7fceea7fc640 1 -- 172.21.15.38:0/772641473 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604629dc400 con 0x560462aecf70 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.280+0000 7fceea7fc640 10 client.15696 handle_command_reply: tid=0 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.280+0000 7fcf0bc69740 1 client.15696 shutdown 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.280+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 msgr2=0x560462a7e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.280+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a7e8d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009e50 tx=0x7fcef40096f0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 shutdown_connections 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560462aecf70 0x560462aef390 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560462a75890 0x560462a7f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560462a70720 0x560462a7e8d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 --2- 172.21.15.38:0/772641473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560462a6fd30 0x560462a7e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fceea7fc640 0 client.15696 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 >> 172.21.15.38:0/772641473 conn(0x560462a6b2a0 msgr2=0x560462a6e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 shutdown_connections 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 1 -- 172.21.15.38:0/772641473 wait complete. 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0bc69740 20 client.15696 trim_cache size 0 max 16384 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:13:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:13:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcef00a2cd0 msgr2=0x7fcef00a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcef00a2cd0 0x7fcef00a5190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcefc09bcc0 tx=0x7fcefc0914a0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 msgr2=0x7fcf0419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 0x7fcf0419ef60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00cce0 tx=0x7fcefc00a620 comp rx=0 tx=0).stop 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 shutdown_connections 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fcef00a2cd0 0x7fcef00a5190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf04102250 0x7fcf0419ef60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf04101890 0x7fcf04197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 --2- 172.21.15.38:0/1769109159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf041006a0 0x7fcf041979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 >> 172.21.15.38:0/1769109159 conn(0x7fcf040fbc10 msgr2=0x7fcf040fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 shutdown_connections 2024-04-22T01:13:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:23.281+0000 7fcf0907d640 1 -- 172.21.15.38:0/1769109159 wait complete. 2024-04-22T01:13:23.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:23 smithi111 ceph-mon[31057]: pgmap v386: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-22T01:13:23.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:23 smithi038 ceph-mon[28043]: pgmap v386: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-22T01:13:23.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:23 smithi203 ceph-mon[31825]: pgmap v386: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-22T01:13:25.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:25 smithi111 ceph-mon[31057]: pgmap v387: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:25 smithi038 ceph-mon[28043]: pgmap v387: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T01:13:25.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:25 smithi203 ceph-mon[31825]: pgmap v387: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T01:13:27.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:27 smithi111 ceph-mon[31057]: pgmap v388: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-22T01:13:27.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:27 smithi038 ceph-mon[28043]: pgmap v388: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-22T01:13:27.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:27 smithi203 ceph-mon[31825]: pgmap v388: 129 pgs: 129 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-22T01:13:29.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:29 smithi111 ceph-mon[31057]: pgmap v389: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-22T01:13:29.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:29 smithi038 ceph-mon[28043]: pgmap v389: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-22T01:13:29.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:29 smithi203 ceph-mon[31825]: pgmap v389: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-22T01:13:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:31 smithi111 ceph-mon[31057]: pgmap v390: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:31.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:31 smithi038 ceph-mon[28043]: pgmap v390: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:31.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:31 smithi203 ceph-mon[31825]: pgmap v390: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:33.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:33 smithi111 ceph-mon[31057]: pgmap v391: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-04-22T01:13:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:33 smithi038 ceph-mon[28043]: pgmap v391: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-04-22T01:13:33.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:33 smithi203 ceph-mon[31825]: pgmap v391: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-04-22T01:13:35.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:35 smithi111 ceph-mon[31057]: pgmap v392: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:35.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:35 smithi038 ceph-mon[28043]: pgmap v392: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:35.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:35 smithi203 ceph-mon[31825]: pgmap v392: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 -- 172.21.15.38:0/556098214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec30102250 msgr2=0x7fec30108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 --2- 172.21.15.38:0/556098214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec30102250 0x7fec30108ae0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fec28009ca0 tx=0x7fec2802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 -- 172.21.15.38:0/556098214 shutdown_connections 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 --2- 172.21.15.38:0/556098214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec30102250 0x7fec30108ae0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 --2- 172.21.15.38:0/556098214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec30101890 0x7fec30101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 --2- 172.21.15.38:0/556098214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec301006a0 0x7fec30100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 -- 172.21.15.38:0/556098214 >> 172.21.15.38:0/556098214 conn(0x7fec300fbc10 msgr2=0x7fec300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 -- 172.21.15.38:0/556098214 shutdown_connections 2024-04-22T01:13:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.382+0000 7fec34888640 1 -- 172.21.15.38:0/556098214 wait complete. 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.383+0000 7fec34888640 1 Processor -- start 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.383+0000 7fec34888640 1 -- start start 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec34888640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 0x7fec301968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec34888640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec30101890 0x7fec30196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec34888640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec30102250 0x7fec3019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec34888640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fec3010c320 con 0x7fec301006a0 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec34888640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec3010c1a0 con 0x7fec30102250 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec34888640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec3010c4a0 con 0x7fec30101890 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 0x7fec301968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 0x7fec301968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37366/0 (socket says 172.21.15.38:37366) 2024-04-22T01:13:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 -- 172.21.15.38:0/3544082241 learned_addr learned my addr 172.21.15.38:0/3544082241 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec30102250 0x7fec3019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec27fff640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec30101890 0x7fec30196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 -- 172.21.15.38:0/3544082241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec30101890 msgr2=0x7fec30196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec30101890 0x7fec30196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 -- 172.21.15.38:0/3544082241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec30102250 msgr2=0x7fec3019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec30102250 0x7fec3019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 -- 172.21.15.38:0/3544082241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3019e500 con 0x7fec301006a0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.384+0000 7fec2e59c640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 0x7fec301968e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fec2000e9c0 tx=0x7fec2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.385+0000 7fec277fe640 1 -- 172.21.15.38:0/3544082241 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2000cd80 con 0x7fec301006a0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.385+0000 7fec277fe640 1 -- 172.21.15.38:0/3544082241 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec200035f0 con 0x7fec301006a0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.385+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec30068e50 con 0x7fec301006a0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.385+0000 7fec277fe640 1 -- 172.21.15.38:0/3544082241 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec20010690 con 0x7fec301006a0 2024-04-22T01:13:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.385+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec30069350 con 0x7fec301006a0 2024-04-22T01:13:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.387+0000 7fec277fe640 1 -- 172.21.15.38:0/3544082241 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fec200037a0 con 0x7fec301006a0 2024-04-22T01:13:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.387+0000 7fec37473740 1 Processor -- start 2024-04-22T01:13:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.387+0000 7fec37473740 1 -- start start 2024-04-22T01:13:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec37473740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da82150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec37473740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 0x55b55da87220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec37473740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da89d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec37473740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b55d9e6b70 con 0x55b55da82720 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec37473740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b55d98f470 con 0x55b55da87890 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec37473740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b55d9e2b30 con 0x55b55da81d30 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec277fe640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fec1c0a2cd0 0x7fec1c0a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 0x55b55da87220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fec1c0a2cd0 0x7fec1c0a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 0x55b55da87220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37370/0 (socket says 172.21.15.38:37370) 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 -- 172.21.15.38:0/675557470 learned_addr learned my addr 172.21.15.38:0/675557470 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec27fff640 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da89d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da82150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec277fe640 1 -- 172.21.15.38:0/3544082241 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fec200cf930 con 0x7fec301006a0 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 -- 172.21.15.38:0/675557470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 msgr2=0x55b55da82150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da82150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 -- 172.21.15.38:0/675557470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 msgr2=0x55b55da89d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da89d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec2e59c640 1 -- 172.21.15.38:0/675557470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55d6fb210 con 0x55b55da82720 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.388+0000 7fec27fff640 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da89d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.389+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fec1c0a2cd0 0x7fec1c0a5190 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fec28005e30 tx=0x7fec28009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.389+0000 7fec2e59c640 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 0x55b55da87220 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fec200058c0 tx=0x7fec20003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66381b5a4272dd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.389+0000 7fec24ff9640 1 -- 172.21.15.38:0/675557470 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2009bc20 con 0x55b55da82720 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.389+0000 7fec24ff9640 1 -- 172.21.15.38:0/675557470 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec2009ffd0 con 0x55b55da82720 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.389+0000 7fec24ff9640 1 -- 172.21.15.38:0/675557470 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2009be80 con 0x55b55da82720 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- 172.21.15.38:0/675557470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 msgr2=0x55b55da87220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 0x55b55da87220 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fec200058c0 tx=0x7fec20003f10 comp rx=0 tx=0).stop 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- 172.21.15.38:0/675557470 shutdown_connections 2024-04-22T01:13:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da89d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da82720 0x55b55da87220 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- 172.21.15.38:0/675557470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da82150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- 172.21.15.38:0/675557470 >> 172.21.15.38:0/675557470 conn(0x55b55da7d2a0 msgr2=0x55b55da7d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- 172.21.15.38:0/675557470 shutdown_connections 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- 172.21.15.38:0/675557470 wait complete. 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- start start 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 Processor -- start 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da8c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da82720 0x55b55da8c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da935d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b55da8a780 con 0x55b55da81d30 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b55d9a2a10 con 0x55b55da87890 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b55da8a900 con 0x55b55da82720 2024-04-22T01:13:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec37473740 10 client.0 _ll_register_callbacks cb 0x55b55d9ddb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.390+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da8c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da8c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37374/0 (socket says 172.21.15.38:37374) 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 -- 172.21.15.38:0/1055218148 learned_addr learned my addr 172.21.15.38:0/1055218148 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da935d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 -- 172.21.15.38:0/1055218148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da82720 msgr2=0x55b55da8c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da82720 0x55b55da8c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 -- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 msgr2=0x55b55da935d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da935d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 -- 172.21.15.38:0/1055218148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b55d7c7620 con 0x55b55da81d30 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da935d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec2e59c640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da8c0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fec2000c6a0 tx=0x7fec20003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8b8e445e24e058 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec15ffb640 1 -- 172.21.15.38:0/1055218148 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec200ba020 con 0x55b55da81d30 2024-04-22T01:13:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec15ffb640 1 -- 172.21.15.38:0/1055218148 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fec200a4400 con 0x55b55da81d30 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec2009ffd0 con 0x55b55da81d30 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec15ffb640 1 -- 172.21.15.38:0/1055218148 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fec2009bc20 con 0x55b55da81d30 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec37473740 10 client.15708 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b55da96940 con 0x55b55da81d30 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.391+0000 7fec15ffb640 1 -- 172.21.15.38:0/1055218148 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec2009ffd0 con 0x55b55da81d30 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec15ffb640 1 -- 172.21.15.38:0/1055218148 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fec200979e0 con 0x55b55da81d30 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec37473740 10 client.15708 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec37473740 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec37473740 20 client.15708 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec37473740 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55da97ac0 0x55b55da99eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec37473740 4 client.15708 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b55d9f40b0 con 0x55b55da97ac0 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec27fff640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55da97ac0 0x55b55da99eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec15ffb640 10 client.15708 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.392+0000 7fec27fff640 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55da97ac0 0x55b55da99eb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23cf1a7554682cd9 server_cookie=c327dd874fe576b1 in_seq=0 out_seq=0 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec15ffb640 1 -- 172.21.15.38:0/1055218148 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b55d9f40b0 con 0x55b55da97ac0 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec15ffb640 10 client.15708 handle_command_reply: tid=0 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 client.15708 shutdown 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 msgr2=0x55b55da8c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da8c0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fec2000c6a0 tx=0x7fec20003d40 comp rx=0 tx=0).stop 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 shutdown_connections 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55da97ac0 0x55b55da99eb0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da87890 0x55b55da935d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da82720 0x55b55da8c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec15ffb640 0 client.15708 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 --2- 172.21.15.38:0/1055218148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da8c0c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 >> 172.21.15.38:0/1055218148 conn(0x55b55da7d2a0 msgr2=0x55b55da7c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.393+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 shutdown_connections 2024-04-22T01:13:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.394+0000 7fec37473740 1 -- 172.21.15.38:0/1055218148 wait complete. 2024-04-22T01:13:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.394+0000 7fec37473740 20 client.15708 trim_cache size 0 max 16384 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 Processor -- start 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 -- start start 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da93590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 0x55b55da80c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fec301a8220 con 0x55b55da87890 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec301a9d60 con 0x55b55da82720 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec37473740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec301a88f0 con 0x55b55da81d30 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 0x55b55da80c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 0x55b55da80c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37378/0 (socket says 172.21.15.38:37378) 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 -- 172.21.15.38:0/3041586329 learned_addr learned my addr 172.21.15.38:0/3041586329 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec27fff640 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da93590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2e59c640 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 -- 172.21.15.38:0/3041586329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 msgr2=0x55b55da8c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da8c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 -- 172.21.15.38:0/3041586329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 msgr2=0x55b55da93590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da93590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 -- 172.21.15.38:0/3041586329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55da7e010 con 0x55b55da87890 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2e59c640 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da8c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.409+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 0x55b55da80c10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fec28009ca0 tx=0x7fec28047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e79a0e84c9b2d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec17fff640 1 -- 172.21.15.38:0/3041586329 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28051070 con 0x55b55da87890 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec17fff640 1 -- 172.21.15.38:0/3041586329 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec28003e50 con 0x55b55da87890 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec17fff640 1 -- 172.21.15.38:0/3041586329 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28039040 con 0x55b55da87890 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 -- 172.21.15.38:0/3041586329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 msgr2=0x55b55da80c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 0x55b55da80c10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fec28009ca0 tx=0x7fec28047000 comp rx=0 tx=0).stop 2024-04-22T01:13:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 -- 172.21.15.38:0/3041586329 shutdown_connections 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da87890 0x55b55da80c10 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da93590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 --2- 172.21.15.38:0/3041586329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da81d30 0x55b55da8c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 -- 172.21.15.38:0/3041586329 >> 172.21.15.38:0/3041586329 conn(0x55b55da7d2a0 msgr2=0x55b55da80500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 -- 172.21.15.38:0/3041586329 shutdown_connections 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.410+0000 7fec37473740 1 -- 172.21.15.38:0/3041586329 wait complete. 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 -- start start 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 Processor -- start 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da90a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da90fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da87890 0x55b55da91fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fec301a9690 con 0x55b55da81d30 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fec301a8fc0 con 0x55b55da82720 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec301a61e0 con 0x55b55da87890 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec37473740 10 client.0 _ll_register_callbacks cb 0x7fec2009cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da90a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da90a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37382/0 (socket says 172.21.15.38:37382) 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec2e59c640 1 -- 172.21.15.38:0/1185411223 learned_addr learned my addr 172.21.15.38:0/1185411223 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec27fff640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da90fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.411+0000 7fec2ed9d640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da87890 0x55b55da91fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec2e59c640 1 -- 172.21.15.38:0/1185411223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da87890 msgr2=0x55b55da91fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec2e59c640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da87890 0x55b55da91fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec2e59c640 1 -- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 msgr2=0x55b55da90fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec2e59c640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da90fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec2e59c640 1 -- 172.21.15.38:0/1185411223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b55da925b0 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec27fff640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da90fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec2e59c640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da90a70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fec2009bbb0 tx=0x7fec200cd120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533c2757d786c36a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec15ffb640 1 -- 172.21.15.38:0/1185411223 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec200ba070 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec15ffb640 1 -- 172.21.15.38:0/1185411223 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fec200a4400 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec28003e50 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec15ffb640 1 -- 172.21.15.38:0/1185411223 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fec2009ffd0 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec37473740 10 client.15714 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b55da925b0 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec15ffb640 1 -- 172.21.15.38:0/1185411223 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec28003e50 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec15ffb640 1 -- 172.21.15.38:0/1185411223 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fec2009b100 con 0x55b55da81d30 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec37473740 10 client.15714 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec37473740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.412+0000 7fec37473740 20 client.15714 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.413+0000 7fec37473740 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55dafe1a0 0x55b55db00590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.413+0000 7fec37473740 4 client.15714 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.413+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b55d9ee400 con 0x55b55dafe1a0 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.413+0000 7fec27fff640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55dafe1a0 0x55b55db00590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.413+0000 7fec27fff640 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55dafe1a0 0x55b55db00590 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85bd6c4d172f09fc server_cookie=682ecc5a101ad320 in_seq=0 out_seq=0 2024-04-22T01:13:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.413+0000 7fec15ffb640 10 client.15714 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec15ffb640 1 -- 172.21.15.38:0/1185411223 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b55d9ee400 con 0x55b55dafe1a0 2024-04-22T01:13:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec15ffb640 10 client.15714 handle_command_reply: tid=0 2024-04-22T01:13:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 client.15714 shutdown 2024-04-22T01:13:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 msgr2=0x55b55da90a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da90a70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fec2009bbb0 tx=0x7fec200cd120 comp rx=0 tx=0).stop 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 shutdown_connections 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b55dafe1a0 0x55b55db00590 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b55da87890 0x55b55da91fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec15ffb640 0 client.15714 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b55da82720 0x55b55da90fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 --2- 172.21.15.38:0/1185411223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b55da81d30 0x55b55da90a70 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 >> 172.21.15.38:0/1185411223 conn(0x55b55da7d2a0 msgr2=0x55b55da80d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 shutdown_connections 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 1 -- 172.21.15.38:0/1185411223 wait complete. 2024-04-22T01:13:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.417+0000 7fec37473740 20 client.15714 trim_cache size 0 max 16384 2024-04-22T01:13:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:13:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:13:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:13:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fec1c0a2cd0 msgr2=0x7fec1c0a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fec1c0a2cd0 0x7fec1c0a5190 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fec28005e30 tx=0x7fec28009800 comp rx=0 tx=0).stop 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 msgr2=0x7fec301968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 0x7fec301968e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fec2000e9c0 tx=0x7fec2000ee80 comp rx=0 tx=0).stop 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 shutdown_connections 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fec1c0a2cd0 0x7fec1c0a5190 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec30102250 0x7fec3019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec30101890 0x7fec30196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 --2- 172.21.15.38:0/3544082241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fec301006a0 0x7fec301968e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 >> 172.21.15.38:0/3544082241 conn(0x7fec300fbc10 msgr2=0x7fec30104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 shutdown_connections 2024-04-22T01:13:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:36.418+0000 7fec34888640 1 -- 172.21.15.38:0/3544082241 wait complete. 2024-04-22T01:13:37.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:37 smithi111 ceph-mon[31057]: pgmap v393: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T01:13:37.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:37 smithi038 ceph-mon[28043]: pgmap v393: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T01:13:37.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:37 smithi203 ceph-mon[31825]: pgmap v393: 129 pgs: 129 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T01:13:39.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:39 smithi111 ceph-mon[31057]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-22T01:13:39.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:39 smithi038 ceph-mon[28043]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-22T01:13:39.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:39 smithi203 ceph-mon[31825]: pgmap v394: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-22T01:13:40.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:13:40.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.577+0000 7f9593008640 1 -- 172.21.15.38:0/863589573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 msgr2=0x7f958c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.577+0000 7f9593008640 1 --2- 172.21.15.38:0/863589573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c100aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9580009900 tx=0x7f958002bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.577+0000 7f9593008640 1 -- 172.21.15.38:0/863589573 shutdown_connections 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.577+0000 7f9593008640 1 --2- 172.21.15.38:0/863589573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c102230 0x7f958c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.577+0000 7f9593008640 1 --2- 172.21.15.38:0/863589573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c101870 0x7f958c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.577+0000 7f9593008640 1 --2- 172.21.15.38:0/863589573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c100aa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- 172.21.15.38:0/863589573 >> 172.21.15.38:0/863589573 conn(0x7f958c0fbbf0 msgr2=0x7f958c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- 172.21.15.38:0/863589573 shutdown_connections 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- 172.21.15.38:0/863589573 wait complete. 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 Processor -- start 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- start start 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c101870 0x7f958c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c102230 0x7f958c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10c2d0 con 0x7f958c100680 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10c150 con 0x7f958c101870 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9593008640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10c450 con 0x7f958c102230 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9590da4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9590da4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52968/0 (socket says 172.21.15.38:52968) 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.578+0000 7f9590da4640 1 -- 172.21.15.38:0/3759144932 learned_addr learned my addr 172.21.15.38:0/3759144932 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f95915a5640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c102230 0x7f958c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f958bfff640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c101870 0x7f958c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9590da4640 1 -- 172.21.15.38:0/3759144932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c102230 msgr2=0x7f958c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9590da4640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c102230 0x7f958c19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9590da4640 1 -- 172.21.15.38:0/3759144932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c101870 msgr2=0x7f958c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9590da4640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c101870 0x7f958c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9590da4640 1 -- 172.21.15.38:0/3759144932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c19f740 con 0x7f958c100680 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f95915a5640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c102230 0x7f958c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f958bfff640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c101870 0x7f958c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9590da4640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c197b20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9580005ab0 tx=0x7f9580005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9589ffb640 1 -- 172.21.15.38:0/3759144932 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580041070 con 0x7f958c100680 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c19f9d0 con 0x7f958c100680 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9589ffb640 1 -- 172.21.15.38:0/3759144932 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95800054e0 con 0x7f958c100680 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.579+0000 7f9589ffb640 1 -- 172.21.15.38:0/3759144932 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580034440 con 0x7f958c100680 2024-04-22T01:13:40.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.580+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c19fec0 con 0x7f958c100680 2024-04-22T01:13:40.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9589ffb640 1 -- 172.21.15.38:0/3759144932 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f9580044070 con 0x7f958c100680 2024-04-22T01:13:40.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 Processor -- start 2024-04-22T01:13:40.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 -- start start 2024-04-22T01:13:40.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bdf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19be4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19be7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556a19b40560 con 0x556a19bdf360 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556a19ae8730 con 0x556a19be4ec0 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9595bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556a19b3c520 con 0x556a19bdfd50 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bdf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bdf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52984/0 (socket says 172.21.15.38:52984) 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 -- 172.21.15.38:0/2342221847 learned_addr learned my addr 172.21.15.38:0/2342221847 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f958bfff640 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19be7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9589ffb640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f957c0a2c20 0x7f957c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f9590da4640 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19be4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 -- 172.21.15.38:0/2342221847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 msgr2=0x556a19be4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19be4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 -- 172.21.15.38:0/2342221847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 msgr2=0x556a19be7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.583+0000 7f95915a5640 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19be7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f95915a5640 1 -- 172.21.15.38:0/2342221847 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a1991e7d0 con 0x556a19bdf360 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f958bfff640 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19be7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f9589ffb640 1 -- 172.21.15.38:0/3759144932 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f95800f1150 con 0x7f958c100680 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f95915a5640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f957c0a2c20 0x7f957c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f95915a5640 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bdf780 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9584010540 tx=0x7f9584010a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32026f9fb3d39933 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f9576ffd640 1 -- 172.21.15.38:0/2342221847 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958401d400 con 0x556a19bdf360 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f9576ffd640 1 -- 172.21.15.38:0/2342221847 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95840134c0 con 0x556a19bdf360 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.584+0000 7f9576ffd640 1 -- 172.21.15.38:0/2342221847 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584011760 con 0x556a19bdf360 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 -- 172.21.15.38:0/2342221847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 msgr2=0x556a19bdf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bdf780 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9584010540 tx=0x7f9584010a00 comp rx=0 tx=0).stop 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f95915a5640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f957c0a2c20 0x7f957c0a50e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9584021000 tx=0x7f958401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 -- 172.21.15.38:0/2342221847 shutdown_connections 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19be7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19be4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2342221847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bdf780 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 -- 172.21.15.38:0/2342221847 >> 172.21.15.38:0/2342221847 conn(0x556a19bd64c0 msgr2=0x556a19bd68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 -- 172.21.15.38:0/2342221847 shutdown_connections 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 -- 172.21.15.38:0/2342221847 wait complete. 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 -- start start 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.585+0000 7f9595bf5740 1 Processor -- start 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19bdf360 0x556a19bde480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bde9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 0x556a19bdaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556a19be7e70 con 0x556a19be4ec0 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556a19afbe00 con 0x556a19bdf360 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556a19be7ff0 con 0x556a19bdfd50 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9595bf5740 10 client.0 _ll_register_callbacks cb 0x556a19b36eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 0x556a19bdaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 0x556a19bdaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52994/0 (socket says 172.21.15.38:52994) 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 -- 172.21.15.38:0/2288022633 learned_addr learned my addr 172.21.15.38:0/2288022633 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f9590da4640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19bdf360 0x556a19bde480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f958bfff640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bde9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 -- 172.21.15.38:0/2288022633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 msgr2=0x556a19bde9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bde9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 -- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19bdf360 msgr2=0x556a19bde480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19bdf360 0x556a19bde480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 -- 172.21.15.38:0/2288022633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a198543f0 con 0x556a19be4ec0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f95915a5640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 0x556a19bdaf20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9584011d30 tx=0x7f9584011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218a6dda083af4b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.586+0000 7f956bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f956bfff640 1 -- 172.21.15.38:0/2288022633 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584034060 con 0x556a19be4ec0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f956bfff640 1 -- 172.21.15.38:0/2288022633 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9584004c10 con 0x556a19be4ec0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95840134c0 con 0x556a19be4ec0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f956bfff640 1 -- 172.21.15.38:0/2288022633 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f958403b050 con 0x556a19be4ec0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 10 client.15726 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a198140f0 con 0x556a19be4ec0 2024-04-22T01:13:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f956bfff640 1 -- 172.21.15.38:0/2288022633 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95840134c0 con 0x556a19be4ec0 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f956bfff640 1 -- 172.21.15.38:0/2288022633 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f958400f040 con 0x556a19be4ec0 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 10 client.15726 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 20 client.15726 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19bf99b0 0x556a19bfbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 4 client.15726 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.587+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a19b4daa0 con 0x556a19bf99b0 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.588+0000 7f9590da4640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19bf99b0 0x556a19bfbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.588+0000 7f956bfff640 10 client.15726 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:40.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.588+0000 7f9590da4640 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19bf99b0 0x556a19bfbda0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5e843ecefe9b7e2 server_cookie=55d179437f9bf6a3 in_seq=0 out_seq=0 2024-04-22T01:13:40.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f956bfff640 1 -- 172.21.15.38:0/2288022633 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a19b4daa0 con 0x556a19bf99b0 2024-04-22T01:13:40.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f956bfff640 10 client.15726 handle_command_reply: tid=0 2024-04-22T01:13:40.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 client.15726 shutdown 2024-04-22T01:13:40.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 msgr2=0x556a19bdaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 0x556a19bdaf20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9584011d30 tx=0x7f9584011d60 comp rx=0 tx=0).stop 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 shutdown_connections 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19bf99b0 0x556a19bfbda0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19be4ec0 0x556a19bdaf20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bde9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 --2- 172.21.15.38:0/2288022633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19bdf360 0x556a19bde480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f956bfff640 0 client.15726 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 >> 172.21.15.38:0/2288022633 conn(0x556a19bd64c0 msgr2=0x556a19bd56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 shutdown_connections 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 1 -- 172.21.15.38:0/2288022633 wait complete. 2024-04-22T01:13:40.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.589+0000 7f9595bf5740 20 client.15726 trim_cache size 0 max 16384 2024-04-22T01:13:40.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 Processor -- start 2024-04-22T01:13:40.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 -- start start 2024-04-22T01:13:40.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bd68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bd4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19bd5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a8350 con 0x556a19bdf360 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a9e90 con 0x556a19be4ec0 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9595bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a8a20 con 0x556a19bdfd50 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9590da4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bd68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9590da4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bd68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53002/0 (socket says 172.21.15.38:53002) 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.608+0000 7f9590da4640 1 -- 172.21.15.38:0/1527138223 learned_addr learned my addr 172.21.15.38:0/1527138223 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f958bfff640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bd4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f95915a5640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19bd5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f9590da4640 1 -- 172.21.15.38:0/1527138223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 msgr2=0x556a19bd4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f9590da4640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bd4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f9590da4640 1 -- 172.21.15.38:0/1527138223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 msgr2=0x556a19bd5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f9590da4640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19bd5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f9590da4640 1 -- 172.21.15.38:0/1527138223 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a19be1a10 con 0x556a19bdf360 2024-04-22T01:13:40.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f95915a5640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19bd5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f958bfff640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bd4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f9590da4640 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bd68a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f95800b99f0 tx=0x7f95800c1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb3837a882e0cc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f95767fc640 1 -- 172.21.15.38:0/1527138223 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958004b040 con 0x556a19bdf360 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f95767fc640 1 -- 172.21.15.38:0/1527138223 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9580005690 con 0x556a19bdf360 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.609+0000 7f95767fc640 1 -- 172.21.15.38:0/1527138223 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95800bb6c0 con 0x556a19bdf360 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 -- 172.21.15.38:0/1527138223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 msgr2=0x556a19bd68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bd68a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f95800b99f0 tx=0x7f95800c1040 comp rx=0 tx=0).stop 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 -- 172.21.15.38:0/1527138223 shutdown_connections 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19bd5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19bd4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 --2- 172.21.15.38:0/1527138223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19bd68a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 -- 172.21.15.38:0/1527138223 >> 172.21.15.38:0/1527138223 conn(0x556a19bd64c0 msgr2=0x556a19be8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 -- 172.21.15.38:0/1527138223 shutdown_connections 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 -- 172.21.15.38:0/1527138223 wait complete. 2024-04-22T01:13:40.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 -- start start 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.610+0000 7f9595bf5740 1 Processor -- start 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19c0ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19c0b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19c0bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a97c0 con 0x556a19bdf360 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a90f0 con 0x556a19be4ec0 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a6310 con 0x556a19bdfd50 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 10 client.0 _ll_register_callbacks cb 0x7f9584020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19c0ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19c0ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53006/0 (socket says 172.21.15.38:53006) 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 -- 172.21.15.38:0/436790266 learned_addr learned my addr 172.21.15.38:0/436790266 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f95915a5640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19c0bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f958bfff640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19c0b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 -- 172.21.15.38:0/436790266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 msgr2=0x556a19c0b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19c0b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 -- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 msgr2=0x556a19c0bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19c0bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 -- 172.21.15.38:0/436790266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a19bdb290 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f958bfff640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19c0b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9590da4640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19c0ab00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f95800bd290 tx=0x7f95800bbaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf6457ea4c1dc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f956bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f956bfff640 1 -- 172.21.15.38:0/436790266 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958004b050 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f956bfff640 1 -- 172.21.15.38:0/436790266 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f95800ee980 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.611+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a19bd1cc0 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f956bfff640 1 -- 172.21.15.38:0/436790266 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95800bd590 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 10 client.15732 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a19c5f9c0 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f956bfff640 1 -- 172.21.15.38:0/436790266 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556a19bd1cc0 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f956bfff640 1 -- 172.21.15.38:0/436790266 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f95800c5d70 con 0x556a19bdf360 2024-04-22T01:13:40.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 10 client.15732 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 20 client.15732 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19c60230 0x556a19c62650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 4 client.15732 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556a19b47df0 con 0x556a19c60230 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.612+0000 7f958bfff640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19c60230 0x556a19c62650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f958bfff640 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19c60230 0x556a19c62650 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac41b85eec5aa658 server_cookie=3d913e3a9ed4f082 in_seq=0 out_seq=0 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f956bfff640 10 client.15732 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f956bfff640 1 -- 172.21.15.38:0/436790266 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556a19b47df0 con 0x556a19c60230 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f956bfff640 10 client.15732 handle_command_reply: tid=0 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 client.15732 shutdown 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 msgr2=0x556a19c0ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19c0ab00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f95800bd290 tx=0x7f95800bbaa0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 shutdown_connections 2024-04-22T01:13:40.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556a19c60230 0x556a19c62650 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a19be4ec0 0x556a19c0bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a19bdfd50 0x556a19c0b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f956bfff640 0 client.15732 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 --2- 172.21.15.38:0/436790266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a19bdf360 0x556a19c0ab00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 >> 172.21.15.38:0/436790266 conn(0x556a19bd64c0 msgr2=0x556a19bd9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 shutdown_connections 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.613+0000 7f9595bf5740 1 -- 172.21.15.38:0/436790266 wait complete. 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9595bf5740 20 client.15732 trim_cache size 0 max 16384 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:13:40.620 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T01:13:40.621 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T01:13:40.621 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:13:40.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f957c0a2c20 msgr2=0x7f957c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f957c0a2c20 0x7f957c0a50e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9584021000 tx=0x7f958401e000 comp rx=0 tx=0).stop 2024-04-22T01:13:40.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 msgr2=0x7f958c197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c197b20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9580005ab0 tx=0x7f9580005dc0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 shutdown_connections 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f957c0a2c20 0x7f957c0a50e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f958c102230 0x7f958c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f958c101870 0x7f958c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 --2- 172.21.15.38:0/3759144932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f958c100680 0x7f958c197b20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 >> 172.21.15.38:0/3759144932 conn(0x7f958c0fbbf0 msgr2=0x7f958c068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 shutdown_connections 2024-04-22T01:13:40.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.614+0000 7f9593008640 1 -- 172.21.15.38:0/3759144932 wait complete. 2024-04-22T01:13:40.627 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T01:13:40.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:13:40.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 -- 172.21.15.38:0/703513264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c101870 msgr2=0x7f880c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 --2- 172.21.15.38:0/703513264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c101870 0x7f880c101cf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009900 tx=0x7f87fc02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:40.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 -- 172.21.15.38:0/703513264 shutdown_connections 2024-04-22T01:13:40.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 --2- 172.21.15.38:0/703513264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c102230 0x7f880c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 --2- 172.21.15.38:0/703513264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c101870 0x7f880c101cf0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 --2- 172.21.15.38:0/703513264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f880c100680 0x7f880c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 -- 172.21.15.38:0/703513264 >> 172.21.15.38:0/703513264 conn(0x7f880c0fbbf0 msgr2=0x7f880c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 -- 172.21.15.38:0/703513264 shutdown_connections 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 -- 172.21.15.38:0/703513264 wait complete. 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.738+0000 7f8813e2e640 1 Processor -- start 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 -- start start 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f880c100680 0x7f880c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c101870 0x7f880c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 0x7f880c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f880c10c370 con 0x7f880c102230 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f880c10c1f0 con 0x7f880c101870 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8813e2e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f880c10c4f0 con 0x7f880c100680 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 0x7f880c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 0x7f880c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53016/0 (socket says 172.21.15.38:53016) 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 -- 172.21.15.38:0/480955196 learned_addr learned my addr 172.21.15.38:0/480955196 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88113c9640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c101870 0x7f880c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f8811bca640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f880c100680 0x7f880c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 -- 172.21.15.38:0/480955196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f880c100680 msgr2=0x7f880c104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f880c100680 0x7f880c104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 -- 172.21.15.38:0/480955196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c101870 msgr2=0x7f880c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c101870 0x7f880c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 -- 172.21.15.38:0/480955196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c103a20 con 0x7f880c102230 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88113c9640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c101870 0x7f880c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.739+0000 7f88123cb640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 0x7f880c103190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f880400cce0 tx=0x7f880400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.740+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880400f450 con 0x7f880c102230 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.740+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880400fad0 con 0x7f880c102230 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.740+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880c1a1bf0 con 0x7f880c102230 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.740+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804018c10 con 0x7f880c102230 2024-04-22T01:13:40.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.740+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880c1a2110 con 0x7f880c102230 2024-04-22T01:13:40.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.741+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f880c068590 con 0x7f880c102230 2024-04-22T01:13:40.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.742+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f880400f5f0 con 0x7f880c102230 2024-04-22T01:13:40.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.742+0000 7f8802ffd640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87f80a2ca0 0x7f87f80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:40.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.743+0000 7f8811bca640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87f80a2ca0 0x7f87f80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:40.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.743+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f88040cfd70 con 0x7f880c102230 2024-04-22T01:13:40.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.743+0000 7f8811bca640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87f80a2ca0 0x7f87f80a5160 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f880c1016d0 tx=0x7f8808008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:40.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:40.750+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f880c068590 con 0x7f880c102230 2024-04-22T01:13:41.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.004+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f880c1a2390 con 0x7f880c102230 2024-04-22T01:13:41.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.005+0000 7f8802ffd640 1 -- 172.21.15.38:0/480955196 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f8804014030 con 0x7f880c102230 2024-04-22T01:13:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:13:41.005 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:13:41.006 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87f80a2ca0 msgr2=0x7f87f80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87f80a2ca0 0x7f87f80a5160 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f880c1016d0 tx=0x7f8808008040 comp rx=0 tx=0).stop 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 msgr2=0x7f880c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 0x7f880c103190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f880400cce0 tx=0x7f880400a620 comp rx=0 tx=0).stop 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 shutdown_connections 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f87f80a2ca0 0x7f87f80a5160 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f880c102230 0x7f880c103190 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f880c101870 0x7f880c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 --2- 172.21.15.38:0/480955196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f880c100680 0x7f880c104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.007+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 >> 172.21.15.38:0/480955196 conn(0x7f880c0fbbf0 msgr2=0x7f880c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.008+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 shutdown_connections 2024-04-22T01:13:41.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.008+0000 7f8813e2e640 1 -- 172.21.15.38:0/480955196 wait complete. 2024-04-22T01:13:41.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T01:13:41.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 -- 172.21.15.38:0/1323630096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 msgr2=0x7f67c41098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 --2- 172.21.15.38:0/1323630096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c41098b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009ca0 tx=0x7f67bc02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 -- 172.21.15.38:0/1323630096 shutdown_connections 2024-04-22T01:13:41.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 --2- 172.21.15.38:0/1323630096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c41098b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 --2- 172.21.15.38:0/1323630096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c4104b30 0x7f67c4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 --2- 172.21.15.38:0/1323630096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c4102200 0x7f67c41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 -- 172.21.15.38:0/1323630096 >> 172.21.15.38:0/1323630096 conn(0x7f67c40fbbf0 msgr2=0x7f67c40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 -- 172.21.15.38:0/1323630096 shutdown_connections 2024-04-22T01:13:41.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.130+0000 7f67cc165640 1 -- 172.21.15.38:0/1323630096 wait complete. 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 Processor -- start 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 -- start start 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c4102200 0x7f67c4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c4104b30 0x7f67c4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f67c410d420 con 0x7f67c4107460 2024-04-22T01:13:41.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f67c410d2a0 con 0x7f67c4104b30 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67cc165640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67c410d5a0 con 0x7f67c4102200 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53030/0 (socket says 172.21.15.38:53030) 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 -- 172.21.15.38:0/1907624318 learned_addr learned my addr 172.21.15.38:0/1907624318 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67c9700640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c4104b30 0x7f67c4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67c9f01640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c4102200 0x7f67c4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 -- 172.21.15.38:0/1907624318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c4102200 msgr2=0x7f67c4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c4102200 0x7f67c4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 -- 172.21.15.38:0/1907624318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c4104b30 msgr2=0x7f67c4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c4104b30 0x7f67c4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.131+0000 7f67ca702640 1 -- 172.21.15.38:0/1907624318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c419f6b0 con 0x7f67c4107460 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.132+0000 7f67ca702640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c419f040 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f67bc037040 tx=0x7f67bc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.132+0000 7f67baffd640 1 -- 172.21.15.38:0/1907624318 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc041020 con 0x7f67c4107460 2024-04-22T01:13:41.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.132+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c419f940 con 0x7f67c4107460 2024-04-22T01:13:41.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.132+0000 7f67baffd640 1 -- 172.21.15.38:0/1907624318 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67bc034d90 con 0x7f67c4107460 2024-04-22T01:13:41.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.132+0000 7f67baffd640 1 -- 172.21.15.38:0/1907624318 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc039040 con 0x7f67c4107460 2024-04-22T01:13:41.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.132+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c419fe30 con 0x7f67c4107460 2024-04-22T01:13:41.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.134+0000 7f67baffd640 1 -- 172.21.15.38:0/1907624318 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f67bc044020 con 0x7f67c4107460 2024-04-22T01:13:41.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 Processor -- start 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 -- start start 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a17460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a1c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a1f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558c0797c530 con 0x558c07a17a30 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558c07924830 con 0x558c07a17040 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67cee2d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c079784f0 con 0x558c07a1cba0 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67c9f01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a1c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67baffd640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f67b00a2c20 0x7f67b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67c9f01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a1c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53044/0 (socket says 172.21.15.38:53044) 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67c9f01640 1 -- 172.21.15.38:0/3724274260 learned_addr learned my addr 172.21.15.38:0/3724274260 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67c9700640 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a1f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.135+0000 7f67ca702640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f67b00a2c20 0x7f67b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a17460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67baffd640 1 -- 172.21.15.38:0/1907624318 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f67bc0f1360 con 0x7f67c4107460 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 -- 172.21.15.38:0/3724274260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 msgr2=0x558c07a17460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a17460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 -- 172.21.15.38:0/3724274260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 msgr2=0x558c07a1c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a1c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 -- 172.21.15.38:0/3724274260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c079482c0 con 0x558c07a1cba0 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67ca702640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f67b00a2c20 0x7f67b00a50e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0bd560 tx=0x7f67bc0a83e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.136+0000 7f67c9700640 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a1f030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67c4198fa0 tx=0x7f67b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a9fd7d87db2cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.139+0000 7f67affff640 1 -- 172.21.15.38:0/3724274260 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4004020 con 0x558c07a1cba0 2024-04-22T01:13:41.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.139+0000 7f67affff640 1 -- 172.21.15.38:0/3724274260 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b4005500 con 0x558c07a1cba0 2024-04-22T01:13:41.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.139+0000 7f67affff640 1 -- 172.21.15.38:0/3724274260 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4004cc0 con 0x558c07a1cba0 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 -- 172.21.15.38:0/3724274260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 msgr2=0x558c07a1f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a1f030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67c4198fa0 tx=0x7f67b40046c0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 -- 172.21.15.38:0/3724274260 shutdown_connections 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a1f030 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a1c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3724274260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a17460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.140+0000 7f67cee2d740 1 -- 172.21.15.38:0/3724274260 >> 172.21.15.38:0/3724274260 conn(0x558c07a125b0 msgr2=0x558c07a129b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.141+0000 7f67cee2d740 1 -- 172.21.15.38:0/3724274260 shutdown_connections 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.141+0000 7f67cee2d740 1 -- 172.21.15.38:0/3724274260 wait complete. 2024-04-22T01:13:41.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.141+0000 7f67cee2d740 1 -- start start 2024-04-22T01:13:41.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.141+0000 7f67cee2d740 1 Processor -- start 2024-04-22T01:13:41.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.141+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17a30 0x558c07a25be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a1cba0 0x558c07a2cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67cee2d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558c07a23f60 con 0x558c07a1cba0 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67cee2d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558c07937f10 con 0x558c07a17040 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67cee2d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c07a240e0 con 0x558c07a17a30 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67cee2d740 10 client.0 _ll_register_callbacks cb 0x558c0768fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a1cba0 0x558c07a2cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a256a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:38360/0 (socket says 172.21.15.38:38360) 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 -- 172.21.15.38:0/1891356918 learned_addr learned my addr 172.21.15.38:0/1891356918 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9f01640 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17a30 0x558c07a25be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 -- 172.21.15.38:0/1891356918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17a30 msgr2=0x558c07a25be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17a30 0x558c07a25be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 -- 172.21.15.38:0/1891356918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a1cba0 msgr2=0x558c07a2cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a1cba0 0x558c07a2cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 -- 172.21.15.38:0/1891356918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c07909c80 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67c9700640 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a256a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f67b4004850 tx=0x7f67b4012450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7686fff35838bcd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.142+0000 7f67ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67ad7fa640 1 -- 172.21.15.38:0/1891356918 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4022030 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67ad7fa640 1 -- 172.21.15.38:0/1891356918 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f67b4023070 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67b4005500 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67ad7fa640 1 -- 172.21.15.38:0/1891356918 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67b4004cb0 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 10 client.25325 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c076903f0 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67ad7fa640 1 -- 172.21.15.38:0/1891356918 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b4005500 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67ad7fa640 1 -- 172.21.15.38:0/1891356918 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f67b401e410 con 0x558c07a17040 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 10 client.25325 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 10 client.25325 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 20 client.25325 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a311d0 0x558c07a33610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 4 client.25325 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.143+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c07989a70 con 0x558c07a311d0 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.144+0000 7f67c9f01640 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a311d0 0x558c07a33610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.144+0000 7f67ad7fa640 10 client.25325 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.144+0000 7f67c9f01640 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a311d0 0x558c07a33610 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=afe9217ca589251e server_cookie=93c1e0fd6c2f24e2 in_seq=0 out_seq=0 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67ad7fa640 1 -- 172.21.15.38:0/1891356918 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c07989a70 con 0x558c07a311d0 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67ad7fa640 10 client.25325 handle_command_reply: tid=0 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 client.25325 shutdown 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 msgr2=0x558c07a256a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a256a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f67b4004850 tx=0x7f67b4012450 comp rx=0 tx=0).stop 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 shutdown_connections 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a311d0 0x558c07a33610 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a1cba0 0x558c07a2cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17a30 0x558c07a25be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 --2- 172.21.15.38:0/1891356918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17040 0x558c07a256a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67ad7fa640 0 client.25325 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:13:41.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 >> 172.21.15.38:0/1891356918 conn(0x558c07a125b0 msgr2=0x558c07a11850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 shutdown_connections 2024-04-22T01:13:41.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 1 -- 172.21.15.38:0/1891356918 wait complete. 2024-04-22T01:13:41.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.145+0000 7f67cee2d740 20 client.25325 trim_cache size 0 max 16384 2024-04-22T01:13:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 Processor -- start 2024-04-22T01:13:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 -- start start 2024-04-22T01:13:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17040 0x558c07a25b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a2cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a1cba0 0x558c07a15f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a8350 con 0x558c07a17a30 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a9e90 con 0x558c07a1cba0 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.159+0000 7f67cee2d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a8a20 con 0x558c07a17040 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a2cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a2cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53064/0 (socket says 172.21.15.38:53064) 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 -- 172.21.15.38:0/213752997 learned_addr learned my addr 172.21.15.38:0/213752997 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9700640 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17040 0x558c07a25b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9700640 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a1cba0 0x558c07a15f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 -- 172.21.15.38:0/213752997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17040 msgr2=0x558c07a25b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17040 0x558c07a25b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 -- 172.21.15.38:0/213752997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a1cba0 msgr2=0x558c07a15f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a1cba0 0x558c07a15f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 -- 172.21.15.38:0/213752997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0775a7d0 con 0x558c07a17a30 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9700640 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a1cba0 0x558c07a15f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67c9f01640 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a2cbb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x558c07a13520 tx=0x7f67c0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b231044c97612795 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67af7fe640 1 -- 172.21.15.38:0/213752997 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0004250 con 0x558c07a17a30 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67af7fe640 1 -- 172.21.15.38:0/213752997 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c00043f0 con 0x558c07a17a30 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.160+0000 7f67af7fe640 1 -- 172.21.15.38:0/213752997 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c001f630 con 0x558c07a17a30 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 -- 172.21.15.38:0/213752997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 msgr2=0x558c07a2cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a2cbb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x558c07a13520 tx=0x7f67c0005f90 comp rx=0 tx=0).stop 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 -- 172.21.15.38:0/213752997 shutdown_connections 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a1cba0 0x558c07a15f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17a30 0x558c07a2cbb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 --2- 172.21.15.38:0/213752997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a17040 0x558c07a25b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 -- 172.21.15.38:0/213752997 >> 172.21.15.38:0/213752997 conn(0x558c07a125b0 msgr2=0x558c07a15810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 -- 172.21.15.38:0/213752997 shutdown_connections 2024-04-22T01:13:41.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 -- 172.21.15.38:0/213752997 wait complete. 2024-04-22T01:13:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 -- start start 2024-04-22T01:13:41.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.161+0000 7f67cee2d740 1 Processor -- start 2024-04-22T01:13:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 0x558c07a29720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17a30 0x558c07a29c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a95740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a97c0 con 0x558c07a17040 2024-04-22T01:13:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a90f0 con 0x558c07a17a30 2024-04-22T01:13:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a6310 con 0x558c07a1cba0 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67cee2d740 10 client.0 _ll_register_callbacks cb 0x7f67c00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 0x558c07a29720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 0x558c07a29720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53066/0 (socket says 172.21.15.38:53066) 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 -- 172.21.15.38:0/3033822423 learned_addr learned my addr 172.21.15.38:0/3033822423 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a95740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9f01640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17a30 0x558c07a29c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 -- 172.21.15.38:0/3033822423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 msgr2=0x558c07a95740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a95740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 -- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17a30 msgr2=0x558c07a29c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17a30 0x558c07a29c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 -- 172.21.15.38:0/3033822423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c07a2cd30 con 0x558c07a17040 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9f01640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17a30 0x558c07a29c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67c9700640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 0x558c07a29720 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f67b401ea70 tx=0x7f67b4012450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78043049d5ed96be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.162+0000 7f67ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67ad7fa640 1 -- 172.21.15.38:0/3033822423 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4022040 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67ad7fa640 1 -- 172.21.15.38:0/3033822423 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f67b4004e50 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67c00043f0 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67ad7fa640 1 -- 172.21.15.38:0/3033822423 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67b403f050 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 10 client.15756 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c07a96ba0 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67ad7fa640 1 -- 172.21.15.38:0/3033822423 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c00043f0 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67ad7fa640 1 -- 172.21.15.38:0/3033822423 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f67b402d780 con 0x558c07a17040 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 10 client.15756 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 10 client.15756 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 20 client.15756 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a97570 0x558c07a99990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 4 client.15756 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:13:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.163+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c07983dc0 con 0x558c07a97570 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67c9f01640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a97570 0x558c07a99990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67c9f01640 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a97570 0x558c07a99990 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46a2b42bad3ba620 server_cookie=29e0d55028264761 in_seq=0 out_seq=0 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67ad7fa640 10 client.15756 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67ad7fa640 1 -- 172.21.15.38:0/3033822423 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c07983dc0 con 0x558c07a97570 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67ad7fa640 10 client.15756 handle_command_reply: tid=0 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67cee2d740 1 client.15756 shutdown 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 msgr2=0x558c07a29720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.164+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 0x558c07a29720 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f67b401ea70 tx=0x7f67b4012450 comp rx=0 tx=0).stop 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 shutdown_connections 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x558c07a97570 0x558c07a99990 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c07a1cba0 0x558c07a95740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c07a17a30 0x558c07a29c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67ad7fa640 0 client.15756 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 --2- 172.21.15.38:0/3033822423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c07a17040 0x558c07a29720 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 >> 172.21.15.38:0/3033822423 conn(0x558c07a125b0 msgr2=0x558c07a259c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 shutdown_connections 2024-04-22T01:13:41.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 1 -- 172.21.15.38:0/3033822423 wait complete. 2024-04-22T01:13:41.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cee2d740 20 client.15756 trim_cache size 0 max 16384 2024-04-22T01:13:41.171 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f67b00a2c20 msgr2=0x7f67b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f67b00a2c20 0x7f67b00a50e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0bd560 tx=0x7f67bc0a83e0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 msgr2=0x7f67c419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c419f040 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f67bc037040 tx=0x7f67bc032f20 comp rx=0 tx=0).stop 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 shutdown_connections 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f67b00a2c20 0x7f67b00a50e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67c4107460 0x7f67c419f040 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67c4104b30 0x7f67c4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 --2- 172.21.15.38:0/1907624318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67c4102200 0x7f67c4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 >> 172.21.15.38:0/1907624318 conn(0x7f67c40fbbf0 msgr2=0x7f67c4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 shutdown_connections 2024-04-22T01:13:41.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.165+0000 7f67cc165640 1 -- 172.21.15.38:0/1907624318 wait complete. 2024-04-22T01:13:41.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 -- 172.21.15.38:0/12126739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c104b30 msgr2=0x7ff60c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 --2- 172.21.15.38:0/12126739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c104b30 0x7ff60c106f20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009900 tx=0x7ff5f402bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 -- 172.21.15.38:0/12126739 shutdown_connections 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 --2- 172.21.15.38:0/12126739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c107460 0x7ff60c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 --2- 172.21.15.38:0/12126739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c104b30 0x7ff60c106f20 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 --2- 172.21.15.38:0/12126739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c102200 0x7ff60c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 -- 172.21.15.38:0/12126739 >> 172.21.15.38:0/12126739 conn(0x7ff60c0fbbf0 msgr2=0x7ff60c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 -- 172.21.15.38:0/12126739 shutdown_connections 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.280+0000 7ff611913640 1 -- 172.21.15.38:0/12126739 wait complete. 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 Processor -- start 2024-04-22T01:13:41.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 -- start start 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 0x7ff60c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c104b30 0x7ff60c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c107460 0x7ff60c1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c10d410 con 0x7ff60c102200 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c10d290 con 0x7ff60c104b30 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff611913640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c10d590 con 0x7ff60c107460 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 0x7ff60c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 0x7ff60c199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53096/0 (socket says 172.21.15.38:53096) 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff60affd640 1 -- 172.21.15.38:0/1661432420 learned_addr learned my addr 172.21.15.38:0/1661432420 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.281+0000 7ff60b7fe640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c107460 0x7ff60c1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff603fff640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c104b30 0x7ff60c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff60affd640 1 -- 172.21.15.38:0/1661432420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c107460 msgr2=0x7ff60c1a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff60affd640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c107460 0x7ff60c1a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff60affd640 1 -- 172.21.15.38:0/1661432420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c104b30 msgr2=0x7ff60c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff60affd640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c104b30 0x7ff60c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff60affd640 1 -- 172.21.15.38:0/1661432420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60c1a1960 con 0x7ff60c102200 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff603fff640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c104b30 0x7ff60c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff60affd640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 0x7ff60c199d50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00e9f0 tx=0x7ff5fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff610911640 1 -- 172.21.15.38:0/1661432420 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc00cd80 con 0x7ff60c102200 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff610911640 1 -- 172.21.15.38:0/1661432420 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc005500 con 0x7ff60c102200 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60c1a1c50 con 0x7ff60c102200 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff610911640 1 -- 172.21.15.38:0/1661432420 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc010690 con 0x7ff60c102200 2024-04-22T01:13:41.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.282+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60c1a2150 con 0x7ff60c102200 2024-04-22T01:13:41.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.284+0000 7ff610911640 1 -- 172.21.15.38:0/1661432420 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7ff5fc0046c0 con 0x7ff60c102200 2024-04-22T01:13:41.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.285+0000 7ff610911640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff5f80a2c20 0x7ff5f80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.285+0000 7ff603fff640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff5f80a2c20 0x7ff5f80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.285+0000 7ff610911640 1 -- 172.21.15.38:0/1661432420 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff5fc0cfac0 con 0x7ff60c102200 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.285+0000 7ff603fff640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff5f80a2c20 0x7ff5f80a50e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4036040 tx=0x7ff5f4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.285+0000 7ff6145dd740 1 Processor -- start 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 -- start start 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106040 0x55dd70106460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 0x55dd7010b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd7010e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dd7006b530 con 0x55dd70106a30 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd70013830 con 0x55dd70106040 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff6145dd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dd700674f0 con 0x55dd7010bba0 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 0x55dd7010b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 0x55dd7010b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53112/0 (socket says 172.21.15.38:53112) 2024-04-22T01:13:41.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 -- 172.21.15.38:0/665783211 learned_addr learned my addr 172.21.15.38:0/665783211 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff60affd640 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106040 0x55dd70106460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 -- 172.21.15.38:0/665783211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 msgr2=0x55dd7010e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd7010e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 -- 172.21.15.38:0/665783211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106040 msgr2=0x55dd70106460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106040 0x55dd70106460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 -- 172.21.15.38:0/665783211 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd700372c0 con 0x55dd70106a30 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.286+0000 7ff603fff640 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 0x55dd7010b530 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4005ec0 tx=0x7ff5f403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c000a877d77193 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff601ffb640 1 -- 172.21.15.38:0/665783211 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40035f0 con 0x55dd70106a30 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff601ffb640 1 -- 172.21.15.38:0/665783211 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f404f070 con 0x55dd70106a30 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff601ffb640 1 -- 172.21.15.38:0/665783211 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f404ab50 con 0x55dd70106a30 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 -- 172.21.15.38:0/665783211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 msgr2=0x55dd7010b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 0x55dd7010b530 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4005ec0 tx=0x7ff5f403c000 comp rx=0 tx=0).stop 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 -- 172.21.15.38:0/665783211 shutdown_connections 2024-04-22T01:13:41.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd7010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106a30 0x55dd7010b530 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 --2- 172.21.15.38:0/665783211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106040 0x55dd70106460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 -- 172.21.15.38:0/665783211 >> 172.21.15.38:0/665783211 conn(0x55dd701015b0 msgr2=0x55dd701019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 -- 172.21.15.38:0/665783211 shutdown_connections 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.287+0000 7ff6145dd740 1 -- 172.21.15.38:0/665783211 wait complete. 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 -- start start 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 Processor -- start 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70110250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70110790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70117760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dd7010eaf0 con 0x55dd70106040 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dd70026f10 con 0x55dd70106a30 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dd7010ec70 con 0x55dd7010bba0 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff6145dd740 10 client.0 _ll_register_callbacks cb 0x55dd6fd7ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.288+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53114/0 (socket says 172.21.15.38:53114) 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 -- 172.21.15.38:0/1797077899 learned_addr learned my addr 172.21.15.38:0/1797077899 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff603fff640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70110790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60b7fe640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70117760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 -- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 msgr2=0x55dd70117760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70117760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 -- 172.21.15.38:0/1797077899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 msgr2=0x55dd70110790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70110790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 -- 172.21.15.38:0/1797077899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd6fff8c80 con 0x55dd70106040 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60b7fe640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70117760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff60affd640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70110250 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0981c0 tx=0x7ff5fc0cd520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7518dea59352094 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff5eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff5eaffd640 1 -- 172.21.15.38:0/1797077899 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc0a01a0 con 0x55dd70106040 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff5eaffd640 1 -- 172.21.15.38:0/1797077899 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff5fc0ba580 con 0x55dd70106040 2024-04-22T01:13:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5f404f070 con 0x55dd70106040 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff5eaffd640 1 -- 172.21.15.38:0/1797077899 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff5fc0a4a60 con 0x55dd70106040 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff6145dd740 10 client.15768 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.289+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dd6fd7f3f0 con 0x55dd70106040 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff5eaffd640 1 -- 172.21.15.38:0/1797077899 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f404f070 con 0x55dd70106040 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff5eaffd640 1 -- 172.21.15.38:0/1797077899 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff5fc086040 con 0x55dd70106040 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff6145dd740 10 client.15768 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff6145dd740 10 client.15768 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff6145dd740 20 client.15768 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff6145dd740 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd7011c100 0x55dd7011e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff6145dd740 4 client.15768 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd70078a70 con 0x55dd7011c100 2024-04-22T01:13:41.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.290+0000 7ff603fff640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd7011c100 0x55dd7011e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.291+0000 7ff603fff640 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd7011c100 0x55dd7011e570 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=40bac09f468e53f4 server_cookie=757cb942c8dc9720 in_seq=0 out_seq=0 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.291+0000 7ff5eaffd640 10 client.15768 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.291+0000 7ff5eaffd640 1 -- 172.21.15.38:0/1797077899 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd70078a70 con 0x55dd7011c100 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.291+0000 7ff5eaffd640 10 client.15768 handle_command_reply: tid=0 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.291+0000 7ff6145dd740 1 client.15768 shutdown 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 msgr2=0x55dd70110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70110250 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0981c0 tx=0x7ff5fc0cd520 comp rx=0 tx=0).stop 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 shutdown_connections 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd7011c100 0x55dd7011e570 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70117760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70110790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 --2- 172.21.15.38:0/1797077899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70110250 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 >> 172.21.15.38:0/1797077899 conn(0x55dd701015b0 msgr2=0x55dd70100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff5eaffd640 0 client.15768 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:13:41.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 shutdown_connections 2024-04-22T01:13:41.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 1 -- 172.21.15.38:0/1797077899 wait complete. 2024-04-22T01:13:41.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.292+0000 7ff6145dd740 20 client.15768 trim_cache size 0 max 16384 2024-04-22T01:13:41.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 Processor -- start 2024-04-22T01:13:41.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 -- start start 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd701106f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70117720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70104f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c1ac6d0 con 0x55dd70106040 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c1ae210 con 0x55dd70106a30 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff6145dd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c1acda0 con 0x55dd7010bba0 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd701106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd701106f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53122/0 (socket says 172.21.15.38:53122) 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.319+0000 7ff60affd640 1 -- 172.21.15.38:0/654969358 learned_addr learned my addr 172.21.15.38:0/654969358 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff603fff640 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70117720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60b7fe640 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70104f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60affd640 1 -- 172.21.15.38:0/654969358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 msgr2=0x55dd70104f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60affd640 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70104f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60affd640 1 -- 172.21.15.38:0/654969358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 msgr2=0x55dd70117720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60affd640 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70117720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60affd640 1 -- 172.21.15.38:0/654969358 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd6fe497d0 con 0x55dd70106040 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff603fff640 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70117720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff60affd640 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd701106f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0a4530 tx=0x7ff5fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd2ec42eddfeacb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff6017fa640 1 -- 172.21.15.38:0/654969358 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc0a8070 con 0x55dd70106040 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff6017fa640 1 -- 172.21.15.38:0/654969358 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc09b050 con 0x55dd70106040 2024-04-22T01:13:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.320+0000 7ff6017fa640 1 -- 172.21.15.38:0/654969358 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc0b9070 con 0x55dd70106040 2024-04-22T01:13:41.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 -- 172.21.15.38:0/654969358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 msgr2=0x55dd701106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd701106f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0a4530 tx=0x7ff5fc004a40 comp rx=0 tx=0).stop 2024-04-22T01:13:41.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 -- 172.21.15.38:0/654969358 shutdown_connections 2024-04-22T01:13:41.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd7010bba0 0x55dd70104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd70106a30 0x55dd70117720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 --2- 172.21.15.38:0/654969358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd701106f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 -- 172.21.15.38:0/654969358 >> 172.21.15.38:0/654969358 conn(0x55dd701015b0 msgr2=0x55dd70104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 -- 172.21.15.38:0/654969358 shutdown_connections 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 -- 172.21.15.38:0/654969358 wait complete. 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.321+0000 7ff6145dd740 1 -- start start 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 Processor -- start 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70114880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd70106a30 0x55dd70114dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd7010bba0 0x55dd70115dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c1adb40 con 0x55dd70106040 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c1ad470 con 0x55dd7010bba0 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c1aa690 con 0x55dd70106a30 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff6145dd740 10 client.0 _ll_register_callbacks cb 0x7ff5fc00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70114880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70114880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53136/0 (socket says 172.21.15.38:53136) 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 -- 172.21.15.38:0/3152088365 learned_addr learned my addr 172.21.15.38:0/3152088365 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60b7fe640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd7010bba0 0x55dd70115dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff603fff640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd70106a30 0x55dd70114dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 -- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd70106a30 msgr2=0x55dd70114dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd70106a30 0x55dd70114dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 -- 172.21.15.38:0/3152088365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd7010bba0 msgr2=0x55dd70115dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd7010bba0 0x55dd70115dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 -- 172.21.15.38:0/3152088365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd701163c0 con 0x55dd70106040 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff603fff640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd70106a30 0x55dd70114dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff60affd640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70114880 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0981c0 tx=0x7ff5fc09c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a3a67e8d812ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.322+0000 7ff5eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff5eaffd640 1 -- 172.21.15.38:0/3152088365 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc0a8070 con 0x55dd70106040 2024-04-22T01:13:41.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff5eaffd640 1 -- 172.21.15.38:0/3152088365 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff5fc097cc0 con 0x55dd70106040 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5fc09b050 con 0x55dd70106040 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff5eaffd640 1 -- 172.21.15.38:0/3152088365 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff5fc0b9030 con 0x55dd70106040 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 10 client.15774 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dd701176c0 con 0x55dd70106040 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff5eaffd640 1 -- 172.21.15.38:0/3152088365 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc09b050 con 0x55dd70106040 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff5eaffd640 1 -- 172.21.15.38:0/3152088365 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff5fc0ab020 con 0x55dd70106040 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 10 client.15774 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 10 client.15774 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 20 client.15774 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd70182790 0x55dd70184b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 4 client.15774 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.323+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd70072dc0 con 0x55dd70182790 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.324+0000 7ff603fff640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd70182790 0x55dd70184b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.324+0000 7ff603fff640 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd70182790 0x55dd70184b80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9bb96755637b0a7e server_cookie=bca85fd031b6327a in_seq=0 out_seq=0 2024-04-22T01:13:41.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.324+0000 7ff5eaffd640 10 client.15774 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.324+0000 7ff5eaffd640 1 -- 172.21.15.38:0/3152088365 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd70072dc0 con 0x55dd70182790 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.324+0000 7ff5eaffd640 10 client.15774 handle_command_reply: tid=0 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.324+0000 7ff6145dd740 1 client.15774 shutdown 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 msgr2=0x55dd70114880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70114880 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0981c0 tx=0x7ff5fc09c4f0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 shutdown_connections 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55dd70182790 0x55dd70184b80 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dd7010bba0 0x55dd70115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd70106a30 0x55dd70114dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 --2- 172.21.15.38:0/3152088365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd70106040 0x55dd70114880 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 >> 172.21.15.38:0/3152088365 conn(0x55dd701015b0 msgr2=0x55dd700fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff5eaffd640 0 client.15774 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 shutdown_connections 2024-04-22T01:13:41.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 1 -- 172.21.15.38:0/3152088365 wait complete. 2024-04-22T01:13:41.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff6145dd740 20 client.15774 trim_cache size 0 max 16384 2024-04-22T01:13:41.331 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:13:41.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff5f80a2c20 msgr2=0x7ff5f80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff5f80a2c20 0x7ff5f80a50e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4036040 tx=0x7ff5f4000bc0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 msgr2=0x7ff60c199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 0x7ff60c199d50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00e9f0 tx=0x7ff5fc00edf0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 shutdown_connections 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff5f80a2c20 0x7ff5f80a50e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c107460 0x7ff60c1a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff60c104b30 0x7ff60c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 --2- 172.21.15.38:0/1661432420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff60c102200 0x7ff60c199d50 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.325+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 >> 172.21.15.38:0/1661432420 conn(0x7ff60c0fbbf0 msgr2=0x7ff60c1019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.326+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 shutdown_connections 2024-04-22T01:13:41.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.326+0000 7ff611913640 1 -- 172.21.15.38:0/1661432420 wait complete. 2024-04-22T01:13:41.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:41 smithi038 ceph-mon[28043]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:13:41.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:41 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/480955196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:41.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T01:13:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:41 smithi111 ceph-mon[31057]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:13:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:41 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/480955196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:41.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 -- 172.21.15.38:0/2016385227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070101870 msgr2=0x7f5070101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 --2- 172.21.15.38:0/2016385227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070101870 0x7f5070101cf0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5064009900 tx=0x7f506402bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:41.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 -- 172.21.15.38:0/2016385227 shutdown_connections 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 --2- 172.21.15.38:0/2016385227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 0x7f5070108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 --2- 172.21.15.38:0/2016385227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070101870 0x7f5070101cf0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 --2- 172.21.15.38:0/2016385227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070100680 0x7f5070100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 -- 172.21.15.38:0/2016385227 >> 172.21.15.38:0/2016385227 conn(0x7f50700fbbf0 msgr2=0x7f50700fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.443+0000 7f5075c0e640 1 -- 172.21.15.38:0/2016385227 shutdown_connections 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.444+0000 7f5075c0e640 1 -- 172.21.15.38:0/2016385227 wait complete. 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.444+0000 7f5075c0e640 1 Processor -- start 2024-04-22T01:13:41.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.444+0000 7f5075c0e640 1 -- start start 2024-04-22T01:13:41.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f5075c0e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 0x7f5070069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f5075c0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070101870 0x7f507019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f5075c0e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 0x7f507019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f5075c0e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f507010c2e0 con 0x7f5070100680 2024-04-22T01:13:41.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f5075c0e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f507010c160 con 0x7f5070102230 2024-04-22T01:13:41.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f5075c0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f507010c460 con 0x7f5070101870 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 0x7f5070069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 0x7f5070069460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53152/0 (socket says 172.21.15.38:53152) 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 -- 172.21.15.38:0/1203851409 learned_addr learned my addr 172.21.15.38:0/1203851409 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506ffff640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 0x7f507019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506effd640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070101870 0x7f507019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 -- 172.21.15.38:0/1203851409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070101870 msgr2=0x7f507019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070101870 0x7f507019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 -- 172.21.15.38:0/1203851409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 msgr2=0x7f507019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 0x7f507019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 -- 172.21.15.38:0/1203851409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507019f700 con 0x7f5070100680 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506effd640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070101870 0x7f507019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506ffff640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 0x7f507019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506f7fe640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 0x7f5070069460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f506000d770 tx=0x7f506000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.445+0000 7f506cff9640 1 -- 172.21.15.38:0/1203851409 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060017070 con 0x7f5070100680 2024-04-22T01:13:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.446+0000 7f506cff9640 1 -- 172.21.15.38:0/1203851409 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5060009c30 con 0x7f5070100680 2024-04-22T01:13:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.446+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507019f9f0 con 0x7f5070100680 2024-04-22T01:13:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.446+0000 7f506cff9640 1 -- 172.21.15.38:0/1203851409 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060003dd0 con 0x7f5070100680 2024-04-22T01:13:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.446+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f507019ff40 con 0x7f5070100680 2024-04-22T01:13:41.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.448+0000 7f506cff9640 1 -- 172.21.15.38:0/1203851409 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f5060003f70 con 0x7f5070100680 2024-04-22T01:13:41.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.448+0000 7f50787f1740 1 Processor -- start 2024-04-22T01:13:41.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 -- start start 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76e9d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76ea2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ea5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506cff9640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50580a2c20 0x7f50580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555f76dfe530 con 0x555f76e9de40 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555f76da6830 con 0x555f76ea2fb0 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f50787f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555f76dfa4f0 con 0x555f76e9d450 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76ea2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506ffff640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50580a2c20 0x7f50580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76ea2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53166/0 (socket says 172.21.15.38:53166) 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 -- 172.21.15.38:0/957466464 learned_addr learned my addr 172.21.15.38:0/957466464 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506effd640 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ea5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506cff9640 1 -- 172.21.15.38:0/1203851409 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f50600cf010 con 0x7f5070100680 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506ffff640 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76e9d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 -- 172.21.15.38:0/957466464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 msgr2=0x555f76e9d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76e9d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 -- 172.21.15.38:0/957466464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 msgr2=0x555f76ea5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ea5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.449+0000 7f506f7fe640 1 -- 172.21.15.38:0/957466464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f76dca2c0 con 0x555f76e9de40 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.450+0000 7f506ffff640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50580a2c20 0x7f50580a50e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x555f76e9e6a0 tx=0x7f50680095b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.450+0000 7f506f7fe640 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76ea2940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5060009610 tx=0x7f5060004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da9c2cc31c1b72a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.450+0000 7f5055ffb640 1 -- 172.21.15.38:0/957466464 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060020070 con 0x555f76e9de40 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.450+0000 7f5055ffb640 1 -- 172.21.15.38:0/957466464 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f506009ba40 con 0x555f76e9de40 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.450+0000 7f5055ffb640 1 -- 172.21.15.38:0/957466464 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600a3fb0 con 0x555f76e9de40 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 -- 172.21.15.38:0/957466464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 msgr2=0x555f76ea2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76ea2940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5060009610 tx=0x7f5060004590 comp rx=0 tx=0).stop 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 -- 172.21.15.38:0/957466464 shutdown_connections 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ea5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76ea2940 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 --2- 172.21.15.38:0/957466464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76e9d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 -- 172.21.15.38:0/957466464 >> 172.21.15.38:0/957466464 conn(0x555f76e945b0 msgr2=0x555f76e949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 -- 172.21.15.38:0/957466464 shutdown_connections 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 -- 172.21.15.38:0/957466464 wait complete. 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 -- start start 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.451+0000 7f50787f1740 1 Processor -- start 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76ea75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9de40 0x555f76ea7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76eaeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555f76ea5f60 con 0x555f76ea2fb0 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555f76db9f10 con 0x555f76e9de40 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555f76ea60e0 con 0x555f76e9d450 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f50787f1740 10 client.0 _ll_register_callbacks cb 0x555f76b11be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76eaeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76eaeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53172/0 (socket says 172.21.15.38:53172) 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 -- 172.21.15.38:0/1476565415 learned_addr learned my addr 172.21.15.38:0/1476565415 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506effd640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9de40 0x555f76ea7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506f7fe640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76ea75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 -- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 msgr2=0x555f76ea75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76ea75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 -- 172.21.15.38:0/1476565415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9de40 msgr2=0x555f76ea7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9de40 0x555f76ea7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 -- 172.21.15.38:0/1476565415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f76d8bc80 con 0x555f76ea2fb0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506effd640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9de40 0x555f76ea7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f506ffff640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76eaeb50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5068010fe0 tx=0x7f5068004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63afbde1bf564f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.452+0000 7f504effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f504effd640 1 -- 172.21.15.38:0/1476565415 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068012e60 con 0x555f76ea2fb0 2024-04-22T01:13:41.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f504effd640 1 -- 172.21.15.38:0/1476565415 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f5068021450 con 0x555f76ea2fb0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f506009ba40 con 0x555f76ea2fb0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f504effd640 1 -- 172.21.15.38:0/1476565415 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5068025070 con 0x555f76ea2fb0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 10 client.15786 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555f76b123f0 con 0x555f76ea2fb0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f504effd640 1 -- 172.21.15.38:0/1476565415 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f506009ba40 con 0x555f76ea2fb0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f504effd640 1 -- 172.21.15.38:0/1476565415 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f506801e040 con 0x555f76ea2fb0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 10 client.15786 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 10 client.15786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 20 client.15786 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76eb34d0 0x555f76eb5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 4 client.15786 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.453+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f76e0ba70 con 0x555f76eb34d0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.454+0000 7f506f7fe640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76eb34d0 0x555f76eb5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.454+0000 7f506f7fe640 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76eb34d0 0x555f76eb5920 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53347cf4e91ed9c4 server_cookie=ec62daf4eaa26e6e in_seq=0 out_seq=0 2024-04-22T01:13:41.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.454+0000 7f504effd640 10 client.15786 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:13:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f504effd640 1 -- 172.21.15.38:0/1476565415 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f76e0ba70 con 0x555f76eb34d0 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f504effd640 10 client.15786 handle_command_reply: tid=0 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 client.15786 shutdown 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 msgr2=0x555f76eaeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76eaeb50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5068010fe0 tx=0x7f5068004ab0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 shutdown_connections 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76eb34d0 0x555f76eb5920 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76eaeb50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9de40 0x555f76ea7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f504effd640 0 client.15786 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 --2- 172.21.15.38:0/1476565415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76ea75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 >> 172.21.15.38:0/1476565415 conn(0x555f76e945b0 msgr2=0x555f76e93850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 shutdown_connections 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 1 -- 172.21.15.38:0/1476565415 wait complete. 2024-04-22T01:13:41.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.455+0000 7f50787f1740 20 client.15786 trim_cache size 0 max 16384 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.482+0000 7f50787f1740 1 Processor -- start 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 -- start start 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9d450 0x555f76ea7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9de40 0x555f76eaeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76e9e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f50701a8460 con 0x555f76ea2fb0 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50701a9fa0 con 0x555f76e9d450 2024-04-22T01:13:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f50787f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50701a8b30 con 0x555f76e9de40 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76e9e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76e9e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53184/0 (socket says 172.21.15.38:53184) 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 -- 172.21.15.38:0/4256292703 learned_addr learned my addr 172.21.15.38:0/4256292703 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506f7fe640 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9d450 0x555f76ea7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 -- 172.21.15.38:0/4256292703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9de40 msgr2=0x555f76eaeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506effd640 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9de40 0x555f76eaeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9de40 0x555f76eaeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 -- 172.21.15.38:0/4256292703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9d450 msgr2=0x555f76ea7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9d450 0x555f76ea7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 -- 172.21.15.38:0/4256292703 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f76bdc7d0 con 0x555f76ea2fb0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506f7fe640 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9d450 0x555f76ea7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.483+0000 7f506ffff640 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76e9e8e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5068010fb0 tx=0x7f5068009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ef0bd238273a51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50557fa640 1 -- 172.21.15.38:0/4256292703 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068012e60 con 0x555f76ea2fb0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50557fa640 1 -- 172.21.15.38:0/4256292703 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5068011e80 con 0x555f76ea2fb0 2024-04-22T01:13:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50557fa640 1 -- 172.21.15.38:0/4256292703 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068021a80 con 0x555f76ea2fb0 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 -- 172.21.15.38:0/4256292703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 msgr2=0x555f76e9e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76e9e8e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5068010fb0 tx=0x7f5068009760 comp rx=0 tx=0).stop 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 -- 172.21.15.38:0/4256292703 shutdown_connections 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76ea2fb0 0x555f76e9e8e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9de40 0x555f76eaeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 --2- 172.21.15.38:0/4256292703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76e9d450 0x555f76ea7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.484+0000 7f50787f1740 1 -- 172.21.15.38:0/4256292703 >> 172.21.15.38:0/4256292703 conn(0x555f76e945b0 msgr2=0x555f76e980a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 -- 172.21.15.38:0/4256292703 shutdown_connections 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 -- 172.21.15.38:0/4256292703 wait complete. 2024-04-22T01:13:41.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 -- start start 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 Processor -- start 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76eabc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76eac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ead1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f50701a98d0 con 0x555f76e9de40 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50701a9200 con 0x555f76ea2fb0 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50701a6420 con 0x555f76e9d450 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f50787f1740 10 client.0 _ll_register_callbacks cb 0x7f5068009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f506effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76eac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f506effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76eac190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53190/0 (socket says 172.21.15.38:53190) 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.485+0000 7f506effd640 1 -- 172.21.15.38:0/395700884 learned_addr learned my addr 172.21.15.38:0/395700884 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506f7fe640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76eabc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506ffff640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ead1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506effd640 1 -- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 msgr2=0x555f76eabc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506effd640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76eabc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506effd640 1 -- 172.21.15.38:0/395700884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 msgr2=0x555f76ead1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506effd640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ead1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506effd640 1 -- 172.21.15.38:0/395700884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f76ead790 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f506effd640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76eac190 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f506402c7b0 tx=0x7f50640045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839e613b178b9e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f504effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f504effd640 1 -- 172.21.15.38:0/395700884 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064005510 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f504effd640 1 -- 172.21.15.38:0/395700884 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f50640046c0 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f76e05dc0 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f504effd640 1 -- 172.21.15.38:0/395700884 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f50640048c0 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f50787f1740 10 client.15792 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.486+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555f76eaeaa0 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f504effd640 1 -- 172.21.15.38:0/395700884 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555f76e05dc0 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f504effd640 1 -- 172.21.15.38:0/395700884 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5064044070 con 0x555f76e9de40 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f50787f1740 10 client.15792 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f50787f1740 10 client.15792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:13:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f50787f1740 20 client.15792 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f50787f1740 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76f19f40 0x555f76f1c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f50787f1740 4 client.15792 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f76d836e0 con 0x555f76f19f40 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f506f7fe640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76f19f40 0x555f76f1c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f504effd640 10 client.15792 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.487+0000 7f506f7fe640 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76f19f40 0x555f76f1c360 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fee3a4772afa88a2 server_cookie=369ac51cfb9b8cb1 in_seq=0 out_seq=0 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f504effd640 1 -- 172.21.15.38:0/395700884 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f76d836e0 con 0x555f76f19f40 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f504effd640 10 client.15792 handle_command_reply: tid=0 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 client.15792 shutdown 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 msgr2=0x555f76eac190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76eac190 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f506402c7b0 tx=0x7f50640045f0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 shutdown_connections 2024-04-22T01:13:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x555f76f19f40 0x555f76f1c360 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f76ea2fb0 0x555f76ead1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f76e9de40 0x555f76eac190 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 --2- 172.21.15.38:0/395700884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555f76e9d450 0x555f76eabc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f504effd640 0 client.15792 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 >> 172.21.15.38:0/395700884 conn(0x555f76e945b0 msgr2=0x555f76e92fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 shutdown_connections 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 1 -- 172.21.15.38:0/395700884 wait complete. 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.488+0000 7f50787f1740 20 client.15792 trim_cache size 0 max 16384 2024-04-22T01:13:41.494 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:13:41.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50580a2c20 msgr2=0x7f50580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50580a2c20 0x7f50580a50e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x555f76e9e6a0 tx=0x7f50680095b0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 msgr2=0x7f5070069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 0x7f5070069460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f506000d770 tx=0x7f506000dc30 comp rx=0 tx=0).stop 2024-04-22T01:13:41.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 shutdown_connections 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50580a2c20 0x7f50580a50e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5070102230 0x7f507019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5070101870 0x7f507019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 --2- 172.21.15.38:0/1203851409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5070100680 0x7f5070069460 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 >> 172.21.15.38:0/1203851409 conn(0x7f50700fbbf0 msgr2=0x7f5070104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 shutdown_connections 2024-04-22T01:13:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.489+0000 7f5075c0e640 1 -- 172.21.15.38:0/1203851409 wait complete. 2024-04-22T01:13:41.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T01:13:41.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:41 smithi203 ceph-mon[31825]: pgmap v395: 129 pgs: 129 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:13:41.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:41 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/480955196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:13:41.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.612+0000 7ff7bce6f640 1 -- 172.21.15.38:0/2344435911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 msgr2=0x7ff7b81045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.612+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/2344435911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b81045f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009900 tx=0x7ff7a802bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:41.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 -- 172.21.15.38:0/2344435911 shutdown_connections 2024-04-22T01:13:41.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/2344435911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b8107460 0x7ff7b81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/2344435911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 0x7ff7b8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/2344435911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b81045f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 -- 172.21.15.38:0/2344435911 >> 172.21.15.38:0/2344435911 conn(0x7ff7b80fbbf0 msgr2=0x7ff7b80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 -- 172.21.15.38:0/2344435911 shutdown_connections 2024-04-22T01:13:41.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.613+0000 7ff7bce6f640 1 -- 172.21.15.38:0/2344435911 wait complete. 2024-04-22T01:13:41.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 Processor -- start 2024-04-22T01:13:41.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 -- start start 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 0x7ff7b8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b8107460 0x7ff7b819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b810d430 con 0x7ff7b8102200 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b810d2b0 con 0x7ff7b8107460 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7bce6f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b810d5b0 con 0x7ff7b8104b30 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53208/0 (socket says 172.21.15.38:53208) 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 -- 172.21.15.38:0/1304647837 learned_addr learned my addr 172.21.15.38:0/1304647837 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b8107460 0x7ff7b819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 0x7ff7b8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 -- 172.21.15.38:0/1304647837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 msgr2=0x7ff7b8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 0x7ff7b8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 -- 172.21.15.38:0/1304647837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b8107460 msgr2=0x7ff7b819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b8107460 0x7ff7b819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b659c640 1 -- 172.21.15.38:0/1304647837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b819f6e0 con 0x7ff7b8102200 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.614+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 0x7ff7b8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.615+0000 7ff7b659c640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b8197ac0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009230 tx=0x7ff7a8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.615+0000 7ff7a77fe640 1 -- 172.21.15.38:0/1304647837 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8041070 con 0x7ff7b8102200 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.615+0000 7ff7a77fe640 1 -- 172.21.15.38:0/1304647837 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7a80054e0 con 0x7ff7b8102200 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.615+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7b819f970 con 0x7ff7b8102200 2024-04-22T01:13:41.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.615+0000 7ff7a77fe640 1 -- 172.21.15.38:0/1304647837 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80346a0 con 0x7ff7b8102200 2024-04-22T01:13:41.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.615+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7b819fe60 con 0x7ff7b8102200 2024-04-22T01:13:41.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7a77fe640 1 -- 172.21.15.38:0/1304647837 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7ff7a8044070 con 0x7ff7b8102200 2024-04-22T01:13:41.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7bfa4b740 1 Processor -- start 2024-04-22T01:13:41.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7bfa4b740 1 -- start start 2024-04-22T01:13:41.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7a77fe640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7a00a2c20 0x7ff7a00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60918460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c6091d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.618+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60920030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c6087d530 con 0x555c60918a30 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c60825830 con 0x555c60918040 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555c608794f0 con 0x555c6091dba0 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7a00a2c20 0x7ff7a00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c6091d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7a77fe640 1 -- 172.21.15.38:0/1304647837 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff7a80f1210 con 0x7ff7b8102200 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c6091d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53210/0 (socket says 172.21.15.38:53210) 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/1960621925 learned_addr learned my addr 172.21.15.38:0/1960621925 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b659c640 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60918460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60920030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/1960621925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 msgr2=0x555c60920030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60920030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/1960621925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 msgr2=0x555c60918460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60918460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/1960621925 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c608492c0 con 0x555c60918a30 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7a00a2c20 0x7ff7a00a50e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0013750 tx=0x7ff7b0013b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.619+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c6091d530 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00bb10 tx=0x7ff7ac00d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960d66043b16305 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7a4ff9640 1 -- 172.21.15.38:0/1960621925 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac017420 con 0x555c60918a30 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7a4ff9640 1 -- 172.21.15.38:0/1960621925 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7ac017aa0 con 0x555c60918a30 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7a4ff9640 1 -- 172.21.15.38:0/1960621925 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac00faa0 con 0x555c60918a30 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/1960621925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 msgr2=0x555c6091d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c6091d530 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00bb10 tx=0x7ff7ac00d340 comp rx=0 tx=0).stop 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/1960621925 shutdown_connections 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60920030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c6091d530 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/1960621925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60918460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/1960621925 >> 172.21.15.38:0/1960621925 conn(0x555c609135b0 msgr2=0x555c609139b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/1960621925 shutdown_connections 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.620+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/1960621925 wait complete. 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 -- start start 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 Processor -- start 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60922220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c60922760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60929730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c60920b50 con 0x555c60918a30 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c60838f10 con 0x555c60918040 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555c60920cd0 con 0x555c6091dba0 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7bfa4b740 10 client.0 _ll_register_callbacks cb 0x555c60590be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c60922760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c60922760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53214/0 (socket says 172.21.15.38:53214) 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/3932172768 learned_addr learned my addr 172.21.15.38:0/3932172768 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60929730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.621+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60922220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/3932172768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 msgr2=0x555c60929730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60929730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 msgr2=0x555c60922220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60922220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b5d9b640 1 -- 172.21.15.38:0/3932172768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c6080ac80 con 0x555c60918a30 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60922220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c60922760 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00bae0 tx=0x7ff7ac017c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08657d3a5ca5614 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff79dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff79dffb640 1 -- 172.21.15.38:0/3932172768 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac027020 con 0x555c60918a30 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff79dffb640 1 -- 172.21.15.38:0/3932172768 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff7ac017830 con 0x555c60918a30 2024-04-22T01:13:41.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7ac017aa0 con 0x555c60918a30 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff79dffb640 1 -- 172.21.15.38:0/3932172768 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff7ac032050 con 0x555c60918a30 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7bfa4b740 10 client.15804 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.622+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c605913f0 con 0x555c60918a30 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff79dffb640 1 -- 172.21.15.38:0/3932172768 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7ac017aa0 con 0x555c60918a30 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff79dffb640 1 -- 172.21.15.38:0/3932172768 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff7ac0239d0 con 0x555c60918a30 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff7bfa4b740 10 client.15804 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff7bfa4b740 10 client.15804 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff7bfa4b740 20 client.15804 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c609300b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff7bfa4b740 4 client.15804 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.623+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c6088aa70 con 0x555c6092dc80 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.624+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c609300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.624+0000 7ff79dffb640 10 client.15804 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.624+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c609300b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2006b9e2b33b972 server_cookie=737787660ad8142b in_seq=0 out_seq=0 2024-04-22T01:13:41.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff79dffb640 1 -- 172.21.15.38:0/3932172768 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c6088aa70 con 0x555c6092dc80 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff79dffb640 10 client.15804 handle_command_reply: tid=0 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 client.15804 shutdown 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 msgr2=0x555c60922760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c60922760 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00bae0 tx=0x7ff7ac017c50 comp rx=0 tx=0).stop 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 shutdown_connections 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c609300b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff79dffb640 0 client.15804 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c6091dba0 0x555c60929730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c60918a30 0x555c60922760 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3932172768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60922220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 >> 172.21.15.38:0/3932172768 conn(0x555c609135b0 msgr2=0x555c60912850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 shutdown_connections 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3932172768 wait complete. 2024-04-22T01:13:41.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.625+0000 7ff7bfa4b740 20 client.15804 trim_cache size 0 max 16384 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.638+0000 7ff7bfa4b740 1 Processor -- start 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.638+0000 7ff7bfa4b740 1 -- start start 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c6091a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c609296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c609126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b81aa480 con 0x555c6091dba0 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b81abfc0 con 0x555c60918040 2024-04-22T01:13:41.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b81aab50 con 0x555c60918a30 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c609126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c609126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53228/0 (socket says 172.21.15.38:53228) 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/2055347606 learned_addr learned my addr 172.21.15.38:0/2055347606 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b659c640 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c6091a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c609296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/2055347606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 msgr2=0x555c609296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c609296f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/2055347606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 msgr2=0x555c6091a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c6091a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/2055347606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c6065b7d0 con 0x555c6091dba0 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c609296f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.639+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c609126d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0013350 tx=0x7ff7b001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0ec9778161332f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.640+0000 7ff79ffff640 1 -- 172.21.15.38:0/2055347606 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0012bd0 con 0x555c6091dba0 2024-04-22T01:13:41.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.640+0000 7ff79ffff640 1 -- 172.21.15.38:0/2055347606 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b0025070 con 0x555c6091dba0 2024-04-22T01:13:41.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.640+0000 7ff79ffff640 1 -- 172.21.15.38:0/2055347606 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0020400 con 0x555c6091dba0 2024-04-22T01:13:41.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.640+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/2055347606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 msgr2=0x555c609126d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.640+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c609126d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0013350 tx=0x7ff7b001c000 comp rx=0 tx=0).stop 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/2055347606 shutdown_connections 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c609126d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c609296f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/2055347606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c6091a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/2055347606 >> 172.21.15.38:0/2055347606 conn(0x555c609135b0 msgr2=0x555c60911fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/2055347606 shutdown_connections 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/2055347606 wait complete. 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- start start 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 Processor -- start 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60927030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c60927570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c60928580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b81ab8f0 con 0x555c6091dba0 2024-04-22T01:13:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b81ab220 con 0x555c60918040 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b81a8440 con 0x555c60918a30 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.641+0000 7ff7bfa4b740 10 client.0 _ll_register_callbacks cb 0x7ff7a8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c60928580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c60928580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53236/0 (socket says 172.21.15.38:53236) 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/3217718612 learned_addr learned my addr 172.21.15.38:0/3217718612 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60927030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b5d9b640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c60927570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/3217718612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 msgr2=0x555c60927570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c60927570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 msgr2=0x555c60927030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60927030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 -- 172.21.15.38:0/3217718612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c60928b70 con 0x555c6091dba0 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7b6d9d640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c60928580 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00077f0 tx=0x7ff7b0007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a2fd177ca1f241 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff79dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff79dffb640 1 -- 172.21.15.38:0/3217718612 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0036060 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff79dffb640 1 -- 172.21.15.38:0/3217718612 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff7b0021400 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7b0025070 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff79dffb640 1 -- 172.21.15.38:0/3217718612 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff7b003d050 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7bfa4b740 10 client.15810 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.642+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c60994330 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff79dffb640 1 -- 172.21.15.38:0/3217718612 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b0025070 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff79dffb640 1 -- 172.21.15.38:0/3217718612 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff7b00055b0 con 0x555c6091dba0 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7bfa4b740 10 client.15810 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7bfa4b740 10 client.15810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7bfa4b740 20 client.15810 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c60996d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7bfa4b740 4 client.15810 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:13:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c60884dc0 con 0x555c6092dc80 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c60996d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff7b659c640 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c60996d50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0c1aa58f22a2f59 server_cookie=423ed3f14c9ee136 in_seq=0 out_seq=0 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.643+0000 7ff79dffb640 10 client.15810 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff79dffb640 1 -- 172.21.15.38:0/3217718612 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c60884dc0 con 0x555c6092dc80 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff79dffb640 10 client.15810 handle_command_reply: tid=0 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 client.15810 shutdown 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 msgr2=0x555c60928580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c60928580 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00077f0 tx=0x7ff7b0007860 comp rx=0 tx=0).stop 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 shutdown_connections 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x555c6092dc80 0x555c60996d50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c6091dba0 0x555c60928580 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff79dffb640 0 client.15810 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c60918a30 0x555c60927570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 --2- 172.21.15.38:0/3217718612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c60918040 0x555c60927030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 >> 172.21.15.38:0/3217718612 conn(0x555c609135b0 msgr2=0x555c60912850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 shutdown_connections 2024-04-22T01:13:41.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 1 -- 172.21.15.38:0/3217718612 wait complete. 2024-04-22T01:13:41.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bfa4b740 20 client.15810 trim_cache size 0 max 16384 2024-04-22T01:13:41.653 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.644+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7a00a2c20 msgr2=0x7ff7a00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7a00a2c20 0x7ff7a00a50e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0013750 tx=0x7ff7b0013b50 comp rx=0 tx=0).stop 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 msgr2=0x7ff7b8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b8197ac0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009230 tx=0x7ff7a8004040 comp rx=0 tx=0).stop 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 shutdown_connections 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff7a00a2c20 0x7ff7a00a50e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b8107460 0x7ff7b819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7b8104b30 0x7ff7b8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 --2- 172.21.15.38:0/1304647837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7b8102200 0x7ff7b8197ac0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 >> 172.21.15.38:0/1304647837 conn(0x7ff7b80fbbf0 msgr2=0x7ff7b8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 shutdown_connections 2024-04-22T01:13:41.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.645+0000 7ff7bce6f640 1 -- 172.21.15.38:0/1304647837 wait complete. 2024-04-22T01:13:41.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 -- 172.21.15.38:0/3175302314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 msgr2=0x7f9bb0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 --2- 172.21.15.38:0/3175302314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0101d10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009ca0 tx=0x7f9ba402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 -- 172.21.15.38:0/3175302314 shutdown_connections 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 --2- 172.21.15.38:0/3175302314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb0102250 0x7f9bb0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 --2- 172.21.15.38:0/3175302314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0101d10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 --2- 172.21.15.38:0/3175302314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb01006a0 0x7f9bb0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 -- 172.21.15.38:0/3175302314 >> 172.21.15.38:0/3175302314 conn(0x7f9bb00fbc30 msgr2=0x7f9bb00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 -- 172.21.15.38:0/3175302314 shutdown_connections 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.763+0000 7f9bb815d640 1 -- 172.21.15.38:0/3175302314 wait complete. 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 Processor -- start 2024-04-22T01:13:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 -- start start 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb01006a0 0x7f9bb01045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb0102250 0x7f9bb01031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010c320 con 0x7f9bb0101890 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010c1a0 con 0x7f9bb0102250 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb815d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010c4a0 con 0x7f9bb01006a0 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53254/0 (socket says 172.21.15.38:53254) 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 -- 172.21.15.38:0/1604959084 learned_addr learned my addr 172.21.15.38:0/1604959084 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb01006a0 0x7f9bb01045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb0102250 0x7f9bb01031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 -- 172.21.15.38:0/1604959084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb01006a0 msgr2=0x7f9bb01045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb01006a0 0x7f9bb01045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 -- 172.21.15.38:0/1604959084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb0102250 msgr2=0x7f9bb01031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb0102250 0x7f9bb01031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb56f8640 1 -- 172.21.15.38:0/1604959084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb01039d0 con 0x7f9bb0101890 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.764+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb0102250 0x7f9bb01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.765+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0102c70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40324c0 tx=0x7f9ba4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.765+0000 7f9ba2ffd640 1 -- 172.21.15.38:0/1604959084 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4039040 con 0x7f9bb0101890 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.765+0000 7f9ba2ffd640 1 -- 172.21.15.38:0/1604959084 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba4035c20 con 0x7f9bb0101890 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.765+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb01a1bd0 con 0x7f9bb0101890 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.765+0000 7f9ba2ffd640 1 -- 172.21.15.38:0/1604959084 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4004310 con 0x7f9bb0101890 2024-04-22T01:13:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.765+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb01a2110 con 0x7f9bb0101890 2024-04-22T01:13:41.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.767+0000 7f9ba2ffd640 1 -- 172.21.15.38:0/1604959084 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f9ba4046030 con 0x7f9bb0101890 2024-04-22T01:13:41.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.768+0000 7f9ba2ffd640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9b9c0a2ca0 0x7f9b9c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.768+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9b9c0a2ca0 0x7f9b9c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.768+0000 7f9ba2ffd640 1 -- 172.21.15.38:0/1604959084 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9ba40f0f00 con 0x7f9bb0101890 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.768+0000 7f9bbae1e740 1 Processor -- start 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.768+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9b9c0a2ca0 0x7f9b9c0a5160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9bb01016f0 tx=0x7f9ba800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 -- start start 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc6b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc70530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 0x55d75dc73030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d75dbd0530 con 0x55d75dc6b040 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d75db78830 con 0x55d75dc6ba30 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d75dbcc4f0 con 0x55d75dc70ba0 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc6b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc6b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53264/0 (socket says 172.21.15.38:53264) 2024-04-22T01:13:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1868471303 learned_addr learned my addr 172.21.15.38:0/1868471303 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc70530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1868471303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 msgr2=0x55d75dc73030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 0x55d75dc73030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1868471303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 msgr2=0x55d75dc70530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc70530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1868471303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75db9c2c0 con 0x55d75dc6b040 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc70530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.769+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc6b460 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80058a0 tx=0x7f9ba8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0de967987a2c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9b9bfff640 1 -- 172.21.15.38:0/1868471303 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8004220 con 0x55d75dc6b040 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9b9bfff640 1 -- 172.21.15.38:0/1868471303 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba80043c0 con 0x55d75dc6b040 2024-04-22T01:13:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9b9bfff640 1 -- 172.21.15.38:0/1868471303 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba801e8f0 con 0x55d75dc6b040 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1868471303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 msgr2=0x55d75dc6b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc6b460 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80058a0 tx=0x7f9ba8013000 comp rx=0 tx=0).stop 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1868471303 shutdown_connections 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 0x55d75dc73030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc70530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1868471303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc6b460 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1868471303 >> 172.21.15.38:0/1868471303 conn(0x55d75dc665b0 msgr2=0x55d75dc669b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.770+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1868471303 shutdown_connections 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1868471303 wait complete. 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 -- start start 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 Processor -- start 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6b040 0x55d75dc751f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 0x55d75dc75730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 0x55d75dc7c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d75dc73b50 con 0x55d75dc6ba30 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d75db8bf10 con 0x55d75dc6b040 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d75dc73cd0 con 0x55d75dc70ba0 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bbae1e740 10 client.0 _ll_register_callbacks cb 0x55d75d8e3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bb56f8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 0x55d75dc75730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bb56f8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 0x55d75dc75730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53278/0 (socket says 172.21.15.38:53278) 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.771+0000 7f9bb56f8640 1 -- 172.21.15.38:0/2967514510 learned_addr learned my addr 172.21.15.38:0/2967514510 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6b040 0x55d75dc751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb56f8640 1 -- 172.21.15.38:0/2967514510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 msgr2=0x55d75dc7c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 0x55d75dc7c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb56f8640 1 -- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6b040 msgr2=0x55d75dc751f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6b040 0x55d75dc751f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb56f8640 1 -- 172.21.15.38:0/2967514510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75db5dc80 con 0x55d75dc6ba30 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6b040 0x55d75dc751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 0x55d75dc75730 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40b9830 tx=0x7f9ba4009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fcfd32f8b5b9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9b997fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9b997fa640 1 -- 172.21.15.38:0/2967514510 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba40d6060 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9b997fa640 1 -- 172.21.15.38:0/2967514510 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9ba40d6200 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ba80043c0 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9b997fa640 1 -- 172.21.15.38:0/2967514510 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ba40c1780 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bbae1e740 10 client.15822 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d75d8e43f0 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.772+0000 7f9b997fa640 1 -- 172.21.15.38:0/2967514510 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba80043c0 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9b997fa640 1 -- 172.21.15.38:0/2967514510 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ba403c030 con 0x55d75dc6ba30 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bbae1e740 10 client.15822 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bbae1e740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bbae1e740 20 client.15822 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dc80f60 0x55d75dc83420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bbae1e740 4 client.15822 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d75dbdda70 con 0x55d75dc80f60 2024-04-22T01:13:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dc80f60 0x55d75dc83420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dc80f60 0x55d75dc83420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1284f4d363f0c59 server_cookie=f157584c7572754e in_seq=0 out_seq=0 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.773+0000 7f9b997fa640 10 client.15822 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9b997fa640 1 -- 172.21.15.38:0/2967514510 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d75dbdda70 con 0x55d75dc80f60 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9b997fa640 10 client.15822 handle_command_reply: tid=0 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 client.15822 shutdown 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 msgr2=0x55d75dc75730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 0x55d75dc75730 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40b9830 tx=0x7f9ba4009950 comp rx=0 tx=0).stop 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 shutdown_connections 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dc80f60 0x55d75dc83420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc70ba0 0x55d75dc7c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6ba30 0x55d75dc75730 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9b997fa640 0 client.15822 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/2967514510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6b040 0x55d75dc751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 >> 172.21.15.38:0/2967514510 conn(0x55d75dc665b0 msgr2=0x55d75dc65850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 shutdown_connections 2024-04-22T01:13:41.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 1 -- 172.21.15.38:0/2967514510 wait complete. 2024-04-22T01:13:41.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.774+0000 7f9bbae1e740 20 client.15822 trim_cache size 0 max 16384 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 Processor -- start 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 -- start start 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc69f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6ba30 0x55d75dc6c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc70ba0 0x55d75dc6c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01ac700 con 0x55d75dc6b040 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01ae240 con 0x55d75dc70ba0 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01acdd0 con 0x55d75dc6ba30 2024-04-22T01:13:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc69f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc69f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53286/0 (socket says 172.21.15.38:53286) 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1865114557 learned_addr learned my addr 172.21.15.38:0/1865114557 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc70ba0 0x55d75dc6c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1865114557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6ba30 msgr2=0x55d75dc6c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6ba30 0x55d75dc6c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6ba30 0x55d75dc6c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1865114557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc70ba0 msgr2=0x55d75dc6c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc70ba0 0x55d75dc6c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 -- 172.21.15.38:0/1865114557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75d9ae7d0 con 0x55d75dc6b040 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb56f8640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6ba30 0x55d75dc6c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc70ba0 0x55d75dc6c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.788+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc69f20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8011f50 tx=0x7f9ba80058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956f1354890238c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/1865114557 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8004080 con 0x55d75dc6b040 2024-04-22T01:13:41.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/1865114557 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba8004220 con 0x55d75dc6b040 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/1865114557 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8003b50 con 0x55d75dc6b040 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1865114557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 msgr2=0x55d75dc69f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc69f20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8011f50 tx=0x7f9ba80058f0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1865114557 shutdown_connections 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc70ba0 0x55d75dc6c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6ba30 0x55d75dc6c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/1865114557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc6b040 0x55d75dc69f20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1865114557 >> 172.21.15.38:0/1865114557 conn(0x55d75dc665b0 msgr2=0x55d75dc69810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1865114557 shutdown_connections 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.789+0000 7f9bbae1e740 1 -- 172.21.15.38:0/1865114557 wait complete. 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 -- start start 2024-04-22T01:13:41.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 Processor -- start 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6b040 0x55d75dc791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc79720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 0x55d75dc7a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01adb70 con 0x55d75dc70ba0 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01ad4a0 con 0x55d75dc6ba30 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb01aa6c0 con 0x55d75dc6b040 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bbae1e740 10 client.0 _ll_register_callbacks cb 0x7f9ba4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bb66fa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 0x55d75dc7a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bb66fa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 0x55d75dc7a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53298/0 (socket says 172.21.15.38:53298) 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bb66fa640 1 -- 172.21.15.38:0/3137415901 learned_addr learned my addr 172.21.15.38:0/3137415901 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.790+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6b040 0x55d75dc791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bb66fa640 1 -- 172.21.15.38:0/3137415901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6b040 msgr2=0x55d75dc791e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6b040 0x55d75dc791e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bb66fa640 1 -- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 msgr2=0x55d75dc79720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc79720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bb66fa640 1 -- 172.21.15.38:0/3137415901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75dc7ad20 con 0x55d75dc70ba0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bb66fa640 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 0x55d75dc7a730 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009f80 tx=0x7f9bac010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdaa9f4bc2ddf12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9b997fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9b997fa640 1 -- 172.21.15.38:0/3137415901 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac016070 con 0x55d75dc70ba0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9b997fa640 1 -- 172.21.15.38:0/3137415901 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9bac005220 con 0x55d75dc70ba0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ba8004220 con 0x55d75dc70ba0 2024-04-22T01:13:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9b997fa640 1 -- 172.21.15.38:0/3137415901 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9bac023050 con 0x55d75dc70ba0 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bbae1e740 10 client.15828 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d75dce7560 con 0x55d75dc70ba0 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.791+0000 7f9b997fa640 1 -- 172.21.15.38:0/3137415901 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba8004220 con 0x55d75dc70ba0 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9b997fa640 1 -- 172.21.15.38:0/3137415901 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9bac00f040 con 0x55d75dc70ba0 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bbae1e740 10 client.15828 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bbae1e740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bbae1e740 20 client.15828 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dce7e20 0x55d75dcea240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bbae1e740 4 client.15828 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d75dbd7dc0 con 0x55d75dce7e20 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dce7e20 0x55d75dcea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9bb5ef9640 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dce7e20 0x55d75dcea240 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca085994ac444226 server_cookie=5b5ac884dcdb69c0 in_seq=0 out_seq=0 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.792+0000 7f9b997fa640 10 client.15828 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9b997fa640 1 -- 172.21.15.38:0/3137415901 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d75dbd7dc0 con 0x55d75dce7e20 2024-04-22T01:13:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9b997fa640 10 client.15828 handle_command_reply: tid=0 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 client.15828 shutdown 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 msgr2=0x55d75dc7a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 0x55d75dc7a730 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009f80 tx=0x7f9bac010800 comp rx=0 tx=0).stop 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 shutdown_connections 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d75dce7e20 0x55d75dcea240 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d75dc70ba0 0x55d75dc7a730 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d75dc6ba30 0x55d75dc79720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 --2- 172.21.15.38:0/3137415901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d75dc6b040 0x55d75dc791e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9b997fa640 0 client.15828 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 >> 172.21.15.38:0/3137415901 conn(0x55d75dc665b0 msgr2=0x55d75dc6a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 shutdown_connections 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 1 -- 172.21.15.38:0/3137415901 wait complete. 2024-04-22T01:13:41.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bbae1e740 20 client.15828 trim_cache size 0 max 16384 2024-04-22T01:13:41.800 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:13:41.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9b9c0a2ca0 msgr2=0x7f9b9c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bb815d640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9b9c0a2ca0 0x7f9b9c0a5160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9bb01016f0 tx=0x7f9ba800b390 comp rx=0 tx=0).stop 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 msgr2=0x7f9bb0102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.793+0000 7f9bb815d640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0102c70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40324c0 tx=0x7f9ba4032f40 comp rx=0 tx=0).stop 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 shutdown_connections 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f9b9c0a2ca0 0x7f9b9c0a5160 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb0102250 0x7f9bb01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb0101890 0x7f9bb0102c70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 --2- 172.21.15.38:0/1604959084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb01006a0 0x7f9bb01045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 >> 172.21.15.38:0/1604959084 conn(0x7f9bb00fbc30 msgr2=0x7f9bb00fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 shutdown_connections 2024-04-22T01:13:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:41.794+0000 7f9bb815d640 1 -- 172.21.15.38:0/1604959084 wait complete. 2024-04-22T01:13:42.805 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T01:13:42.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:13:42.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 -- 172.21.15.38:0/2027090975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 msgr2=0x7faa74101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 --2- 172.21.15.38:0/2027090975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa74101cf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009900 tx=0x7faa5c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 -- 172.21.15.38:0/2027090975 shutdown_connections 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 --2- 172.21.15.38:0/2027090975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa74102230 0x7faa74108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 --2- 172.21.15.38:0/2027090975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa74101cf0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 --2- 172.21.15.38:0/2027090975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa74100680 0x7faa74100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 -- 172.21.15.38:0/2027090975 >> 172.21.15.38:0/2027090975 conn(0x7faa740fbbf0 msgr2=0x7faa740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:42.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 -- 172.21.15.38:0/2027090975 shutdown_connections 2024-04-22T01:13:42.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.912+0000 7faa78b3e640 1 -- 172.21.15.38:0/2027090975 wait complete. 2024-04-22T01:13:42.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.913+0000 7faa78b3e640 1 Processor -- start 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.913+0000 7faa78b3e640 1 -- start start 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa74100680 0x7faa74197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa741980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa74102230 0x7faa7419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa7410c3a0 con 0x7faa74101870 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa7410c220 con 0x7faa74102230 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa7410c520 con 0x7faa74100680 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa741980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa741980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53318/0 (socket says 172.21.15.38:53318) 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 -- 172.21.15.38:0/1658519666 learned_addr learned my addr 172.21.15.38:0/1658519666 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa72d9d640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa74102230 0x7faa7419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa7259c640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa74100680 0x7faa74197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 -- 172.21.15.38:0/1658519666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa74100680 msgr2=0x7faa74197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa74100680 0x7faa74197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 -- 172.21.15.38:0/1658519666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa74102230 msgr2=0x7faa7419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa74102230 0x7faa7419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 -- 172.21.15.38:0/1658519666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7419f790 con 0x7faa74101870 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa69d9b640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa741980b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009230 tx=0x7faa5c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa6bfff640 1 -- 172.21.15.38:0/1658519666 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c041070 con 0x7faa74101870 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7419fa20 con 0x7faa74101870 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.914+0000 7faa6bfff640 1 -- 172.21.15.38:0/1658519666 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa5c0054e0 con 0x7faa74101870 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.915+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7419ff60 con 0x7faa74101870 2024-04-22T01:13:42.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.915+0000 7faa6bfff640 1 -- 172.21.15.38:0/1658519666 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c04b060 con 0x7faa74101870 2024-04-22T01:13:42.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.917+0000 7faa6bfff640 1 -- 172.21.15.38:0/1658519666 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7faa5c044070 con 0x7faa74101870 2024-04-22T01:13:42.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.917+0000 7faa7b806740 1 Processor -- start 2024-04-22T01:13:42.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 -- start start 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x5559048201d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x5559048252a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904827da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555904780170 con 0x55590481fdb0 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5559047287c0 con 0x5559048207a0 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7b806740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55590477c130 con 0x555904825910 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa72d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x5559048201d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa72d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x5559048201d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53324/0 (socket says 172.21.15.38:53324) 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa72d9d640 1 -- 172.21.15.38:0/2563900765 learned_addr learned my addr 172.21.15.38:0/2563900765 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa7259c640 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x5559048252a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa6bfff640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa600a2c20 0x7faa600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa69d9b640 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904827da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa72d9d640 1 -- 172.21.15.38:0/2563900765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 msgr2=0x555904827da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa6bfff640 1 -- 172.21.15.38:0/1658519666 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7faa5c0f11b0 con 0x7faa74101870 2024-04-22T01:13:42.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.918+0000 7faa72d9d640 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904827da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa72d9d640 1 -- 172.21.15.38:0/2563900765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 msgr2=0x5559048252a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa72d9d640 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x5559048252a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa72d9d640 1 -- 172.21.15.38:0/2563900765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555904739e60 con 0x55590481fdb0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa7259c640 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x5559048252a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa72d9d640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa600a2c20 0x7faa600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa72d9d640 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x5559048201d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7faa6c010510 tx=0x7faa6c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d58395400ded0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa68d99640 1 -- 172.21.15.38:0/2563900765 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c023070 con 0x55590481fdb0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa68d99640 1 -- 172.21.15.38:0/2563900765 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c0134c0 con 0x55590481fdb0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa68d99640 1 -- 172.21.15.38:0/2563900765 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c011940 con 0x55590481fdb0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.919+0000 7faa72d9d640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa600a2c20 0x7faa600a50e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faa6c020000 tx=0x7faa6c01b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 -- 172.21.15.38:0/2563900765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 msgr2=0x5559048201d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x5559048201d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7faa6c010510 tx=0x7faa6c009f90 comp rx=0 tx=0).stop 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 -- 172.21.15.38:0/2563900765 shutdown_connections 2024-04-22T01:13:42.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904827da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x5559048252a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 --2- 172.21.15.38:0/2563900765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x5559048201d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 -- 172.21.15.38:0/2563900765 >> 172.21.15.38:0/2563900765 conn(0x5559047e34f0 msgr2=0x555904817310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 -- 172.21.15.38:0/2563900765 shutdown_connections 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 -- 172.21.15.38:0/2563900765 wait complete. 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 -- start start 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.920+0000 7faa7b806740 1 Processor -- start 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x55590482a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x55590482a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904831880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5559048288c0 con 0x55590481fdb0 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55590473bd40 con 0x5559048207a0 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555904828a40 con 0x555904825910 2024-04-22T01:13:42.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7b806740 10 client.0 _ll_register_callbacks cb 0x555904777820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x55590482a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x55590482a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53340/0 (socket says 172.21.15.38:53340) 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 -- 172.21.15.38:0/3321412685 learned_addr learned my addr 172.21.15.38:0/3321412685 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa69d9b640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x55590482a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa72d9d640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904831880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 -- 172.21.15.38:0/3321412685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 msgr2=0x555904831880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904831880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 -- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 msgr2=0x55590482a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x55590482a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 -- 172.21.15.38:0/3321412685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590455c7e0 con 0x55590481fdb0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa69d9b640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x55590482a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa72d9d640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904831880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa7259c640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x55590482a300 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7faa6400d240 tx=0x7faa6400d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b41aeba3da0a61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:42.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.921+0000 7faa59ffb640 1 -- 172.21.15.38:0/3321412685 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6401c060 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa59ffb640 1 -- 172.21.15.38:0/3321412685 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7faa64015750 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa6c0134c0 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa59ffb640 1 -- 172.21.15.38:0/3321412685 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7faa64023050 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 10 client.15840 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5559045bcf70 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa59ffb640 1 -- 172.21.15.38:0/3321412685 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c0134c0 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa59ffb640 1 -- 172.21.15.38:0/3321412685 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7faa64017ba0 con 0x55590481fdb0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 10 client.15840 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 20 client.15840 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x555904835e60 0x555904838300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 4 client.15840 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.922+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555904754450 con 0x555904835e60 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.923+0000 7faa69d9b640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x555904835e60 0x555904838300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.923+0000 7faa69d9b640 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x555904835e60 0x555904838300 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66ce6515558be50 server_cookie=ec36dbf4da6db841 in_seq=0 out_seq=0 2024-04-22T01:13:42.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.923+0000 7faa59ffb640 10 client.15840 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa59ffb640 1 -- 172.21.15.38:0/3321412685 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555904754450 con 0x555904835e60 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa59ffb640 10 client.15840 handle_command_reply: tid=0 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 client.15840 shutdown 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 msgr2=0x55590482a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x55590482a300 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7faa6400d240 tx=0x7faa6400d700 comp rx=0 tx=0).stop 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 shutdown_connections 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x555904835e60 0x555904838300 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x555904831880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5559048207a0 0x55590482a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 --2- 172.21.15.38:0/3321412685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590481fdb0 0x55590482a300 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa59ffb640 0 client.15840 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 >> 172.21.15.38:0/3321412685 conn(0x5559047e34f0 msgr2=0x5559048164f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:42.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 shutdown_connections 2024-04-22T01:13:42.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 1 -- 172.21.15.38:0/3321412685 wait complete. 2024-04-22T01:13:42.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.924+0000 7faa7b806740 20 client.15840 trim_cache size 0 max 16384 2024-04-22T01:13:42.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.942+0000 7faa7b806740 1 Processor -- start 2024-04-22T01:13:42.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.942+0000 7faa7b806740 1 -- start start 2024-04-22T01:13:42.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.942+0000 7faa7b806740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590481fdb0 0x55590481ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.942+0000 7faa7b806740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x555904831840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.942+0000 7faa7b806740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555904825910 0x55590482a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa7b806740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa741a8340 con 0x5559048207a0 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa7b806740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa741a9e80 con 0x555904825910 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa7b806740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa741a8a10 con 0x55590481fdb0 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x555904831840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x555904831840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53342/0 (socket says 172.21.15.38:53342) 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 -- 172.21.15.38:0/1640279722 learned_addr learned my addr 172.21.15.38:0/1640279722 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa72d9d640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555904825910 0x55590482a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa7259c640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590481fdb0 0x55590481ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 -- 172.21.15.38:0/1640279722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590481fdb0 msgr2=0x55590481ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590481fdb0 0x55590481ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 -- 172.21.15.38:0/1640279722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555904825910 msgr2=0x55590482a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555904825910 0x55590482a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 -- 172.21.15.38:0/1640279722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555904454170 con 0x5559048207a0 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa7259c640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590481fdb0 0x55590481ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa72d9d640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555904825910 0x55590482a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa69d9b640 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x555904831840 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7faa5c02cfd0 tx=0x7faa5c0a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a7105653a9639b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa5bfff640 1 -- 172.21.15.38:0/1640279722 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0bd130 con 0x5559048207a0 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa5bfff640 1 -- 172.21.15.38:0/1640279722 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa5c005690 con 0x5559048207a0 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.943+0000 7faa5bfff640 1 -- 172.21.15.38:0/1640279722 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0bcf50 con 0x5559048207a0 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 -- 172.21.15.38:0/1640279722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 msgr2=0x555904831840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x555904831840 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7faa5c02cfd0 tx=0x7faa5c0a8040 comp rx=0 tx=0).stop 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 -- 172.21.15.38:0/1640279722 shutdown_connections 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555904825910 0x55590482a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x555904831840 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 --2- 172.21.15.38:0/1640279722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590481fdb0 0x55590481ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 -- 172.21.15.38:0/1640279722 >> 172.21.15.38:0/1640279722 conn(0x5559047e34f0 msgr2=0x555904815c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 -- 172.21.15.38:0/1640279722 shutdown_connections 2024-04-22T01:13:42.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 -- 172.21.15.38:0/1640279722 wait complete. 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.944+0000 7faa7b806740 1 -- start start 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 Processor -- start 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590481fdb0 0x55590482f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x55590482f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x5559048306f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa741a97b0 con 0x5559048207a0 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa741a90e0 con 0x55590481fdb0 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa741a6300 con 0x555904825910 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7b806740 10 client.0 _ll_register_callbacks cb 0x7faa6400b500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x55590482f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x55590482f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53344/0 (socket says 172.21.15.38:53344) 2024-04-22T01:13:42.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 -- 172.21.15.38:0/543989880 learned_addr learned my addr 172.21.15.38:0/543989880 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7259c640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590481fdb0 0x55590482f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa72d9d640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x5559048306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 -- 172.21.15.38:0/543989880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 msgr2=0x5559048306f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x5559048306f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 -- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590481fdb0 msgr2=0x55590482f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590481fdb0 0x55590482f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 -- 172.21.15.38:0/543989880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555904830ce0 con 0x5559048207a0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa7259c640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590481fdb0 0x55590482f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa72d9d640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x5559048306f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa69d9b640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x55590482f6e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0bd580 tx=0x7faa5c0bcf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5acaff001f2c5b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.945+0000 7faa59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa59ffb640 1 -- 172.21.15.38:0/543989880 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0d5070 con 0x5559048207a0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa59ffb640 1 -- 172.21.15.38:0/543989880 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7faa5c0b9c60 con 0x5559048207a0 2024-04-22T01:13:42.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55590478d6b0 con 0x5559048207a0 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa59ffb640 1 -- 172.21.15.38:0/543989880 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7faa5c0bd5f0 con 0x5559048207a0 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 10 client.15846 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55590489c620 con 0x5559048207a0 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa59ffb640 1 -- 172.21.15.38:0/543989880 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55590478d6b0 con 0x5559048207a0 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa59ffb640 1 -- 172.21.15.38:0/543989880 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7faa5c0035f0 con 0x5559048207a0 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 10 client.15846 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 20 client.15846 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55590489cf20 0x55590489f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 4 client.15846 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.946+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555904787a00 con 0x55590489cf20 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.947+0000 7faa7259c640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55590489cf20 0x55590489f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.947+0000 7faa59ffb640 10 client.15846 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:42.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.947+0000 7faa7259c640 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55590489cf20 0x55590489f340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb1f52da9e76d170 server_cookie=de72b38cff0f964 in_seq=0 out_seq=0 2024-04-22T01:13:42.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa59ffb640 1 -- 172.21.15.38:0/543989880 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555904787a00 con 0x55590489cf20 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa59ffb640 10 client.15846 handle_command_reply: tid=0 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 client.15846 shutdown 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 msgr2=0x55590482f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x55590482f6e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0bd580 tx=0x7faa5c0bcf50 comp rx=0 tx=0).stop 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 shutdown_connections 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55590489cf20 0x55590489f340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555904825910 0x5559048306f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5559048207a0 0x55590482f6e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 --2- 172.21.15.38:0/543989880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590481fdb0 0x55590482f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 >> 172.21.15.38:0/543989880 conn(0x5559047e34f0 msgr2=0x5559048164f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa59ffb640 0 client.15846 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 shutdown_connections 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.948+0000 7faa7b806740 1 -- 172.21.15.38:0/543989880 wait complete. 2024-04-22T01:13:42.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa7b806740 20 client.15846 trim_cache size 0 max 16384 2024-04-22T01:13:42.955 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:13:42.955 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T01:13:42.955 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "72afd7cb-73ab-4511-b7ef-e960ebcda3ae", 2024-04-22T01:13:42.955 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T01:13:42.955 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa600a2c20 msgr2=0x7faa600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa600a2c20 0x7faa600a50e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faa6c020000 tx=0x7faa6c01b000 comp rx=0 tx=0).stop 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 msgr2=0x7faa741980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa741980b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009230 tx=0x7faa5c005dd0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 shutdown_connections 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7faa600a2c20 0x7faa600a50e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa74102230 0x7faa7419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa74101870 0x7faa741980b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 --2- 172.21.15.38:0/1658519666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa74100680 0x7faa74197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:42.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 >> 172.21.15.38:0/1658519666 conn(0x7faa740fbbf0 msgr2=0x7faa740fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:42.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 shutdown_connections 2024-04-22T01:13:42.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:42.949+0000 7faa78b3e640 1 -- 172.21.15.38:0/1658519666 wait complete. 2024-04-22T01:13:42.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:13:42.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4080124030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 msgr2=0x7f6d18101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4080124030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18101cf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009900 tx=0x7f6d0402bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4080124030 shutdown_connections 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4080124030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 0x7f6d18108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4080124030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18101cf0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4080124030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d18100680 0x7f6d18100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.069+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4080124030 >> 172.21.15.38:0/4080124030 conn(0x7f6d180fbbf0 msgr2=0x7f6d180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4080124030 shutdown_connections 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4080124030 wait complete. 2024-04-22T01:13:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 Processor -- start 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 -- start start 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d18100680 0x7f6d18104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 0x7f6d18103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810c300 con 0x7f6d18101870 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810c180 con 0x7f6d18102230 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d1eb9a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810c480 con 0x7f6d18100680 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53356/0 (socket says 172.21.15.38:53356) 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.070+0000 7f6d17fff640 1 -- 172.21.15.38:0/4235243368 learned_addr learned my addr 172.21.15.38:0/4235243368 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d1c936640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d18100680 0x7f6d18104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d1d137640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 0x7f6d18103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d17fff640 1 -- 172.21.15.38:0/4235243368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d18100680 msgr2=0x7f6d18104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d17fff640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d18100680 0x7f6d18104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d17fff640 1 -- 172.21.15.38:0/4235243368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 msgr2=0x7f6d18103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d17fff640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 0x7f6d18103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d17fff640 1 -- 172.21.15.38:0/4235243368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d18103a20 con 0x7f6d18101870 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d1d137640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 0x7f6d18103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d17fff640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18102c50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6d0402ce70 tx=0x7f6d04005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d15ffb640 1 -- 172.21.15.38:0/4235243368 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04041070 con 0x7f6d18101870 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d15ffb640 1 -- 172.21.15.38:0/4235243368 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d040054e0 con 0x7f6d18101870 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d181a1c50 con 0x7f6d18101870 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d15ffb640 1 -- 172.21.15.38:0/4235243368 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d040346a0 con 0x7f6d18101870 2024-04-22T01:13:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.071+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d181a21a0 con 0x7f6d18101870 2024-04-22T01:13:43.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d15ffb640 1 -- 172.21.15.38:0/4235243368 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f6d04044070 con 0x7f6d18101870 2024-04-22T01:13:43.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 Processor -- start 2024-04-22T01:13:43.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 -- start start 2024-04-22T01:13:43.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c4d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c52440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c52ab0 0x55dec7c54f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dec7bb2560 con 0x55dec7c4d940 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dec7b5a730 con 0x55dec7c4cf50 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d21785740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dec7bae520 con 0x55dec7c52ab0 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d1c936640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c52440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d1c936640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c52440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53368/0 (socket says 172.21.15.38:53368) 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d1c936640 1 -- 172.21.15.38:0/2348491794 learned_addr learned my addr 172.21.15.38:0/2348491794 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d17fff640 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c52ab0 0x55dec7c54f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.074+0000 7f6d1d137640 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c4d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d15ffb640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6d080a2ca0 0x7f6d080a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1c936640 1 -- 172.21.15.38:0/2348491794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c52ab0 msgr2=0x55dec7c54f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1c936640 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c52ab0 0x55dec7c54f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1d137640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6d080a2ca0 0x7f6d080a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1c936640 1 -- 172.21.15.38:0/2348491794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 msgr2=0x55dec7c4d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1c936640 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c4d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1c936640 1 -- 172.21.15.38:0/2348491794 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec79907d0 con 0x55dec7c4d940 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d15ffb640 1 -- 172.21.15.38:0/4235243368 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6d040f1110 con 0x7f6d18101870 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1c936640 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c52440 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00bcb0 tx=0x7f6d0c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86159437cd411eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d02ffd640 1 -- 172.21.15.38:0/2348491794 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c005020 con 0x55dec7c4d940 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d02ffd640 1 -- 172.21.15.38:0/2348491794 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d0c0047a0 con 0x55dec7c4d940 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d1d137640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6d080a2ca0 0x7f6d080a5160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55dec7c4e1a0 tx=0x7f6d10004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.075+0000 7f6d02ffd640 1 -- 172.21.15.38:0/2348491794 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c011780 con 0x55dec7c4d940 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 -- 172.21.15.38:0/2348491794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 msgr2=0x55dec7c52440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c52440 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00bcb0 tx=0x7f6d0c0107a0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 -- 172.21.15.38:0/2348491794 shutdown_connections 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c52ab0 0x55dec7c54f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c52440 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 --2- 172.21.15.38:0/2348491794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c4d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 -- 172.21.15.38:0/2348491794 >> 172.21.15.38:0/2348491794 conn(0x55dec7c484c0 msgr2=0x55dec7c488c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 -- 172.21.15.38:0/2348491794 shutdown_connections 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 -- 172.21.15.38:0/2348491794 wait complete. 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 -- start start 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.076+0000 7f6d21785740 1 Processor -- start 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c574d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c57a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 0x55dec7c5ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dec7c55a60 con 0x55dec7c52ab0 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dec7b6de00 con 0x55dec7c4cf50 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dec7c55be0 con 0x55dec7c4d940 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d21785740 10 client.0 _ll_register_callbacks cb 0x55dec7ba8eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 0x55dec7c5ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 0x55dec7c5ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53382/0 (socket says 172.21.15.38:53382) 2024-04-22T01:13:43.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 -- 172.21.15.38:0/2408850416 learned_addr learned my addr 172.21.15.38:0/2408850416 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d17fff640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c57a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1c936640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 -- 172.21.15.38:0/2408850416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 msgr2=0x55dec7c57a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c57a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 -- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 msgr2=0x55dec7c574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.077+0000 7f6d1d137640 1 -- 172.21.15.38:0/2408850416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec78c63f0 con 0x55dec7c52ab0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d1d137640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 0x55dec7c5ea50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6d10003ba0 tx=0x7f6d10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d900a29c10e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6cf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6cf7fff640 1 -- 172.21.15.38:0/2408850416 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1001c050 con 0x55dec7c52ab0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d0c0047a0 con 0x55dec7c52ab0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6cf7fff640 1 -- 172.21.15.38:0/2408850416 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6d10004440 con 0x55dec7c52ab0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6cf7fff640 1 -- 172.21.15.38:0/2408850416 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d10004640 con 0x55dec7c52ab0 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 10 client.15858 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:43.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dec78860f0 con 0x55dec7c52ab0 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6cf7fff640 1 -- 172.21.15.38:0/2408850416 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d0c0047a0 con 0x55dec7c52ab0 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6cf7fff640 1 -- 172.21.15.38:0/2408850416 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6d1000f4b0 con 0x55dec7c52ab0 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 10 client.15858 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 20 client.15858 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7c63420 0x55dec7c658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 4 client.15858 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.078+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dec7bbfaa0 con 0x55dec7c63420 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.079+0000 7f6d1c936640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7c63420 0x55dec7c658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.079+0000 7f6d1c936640 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7c63420 0x55dec7c658e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ff2c08774ef70 server_cookie=da9a19680b8f00c8 in_seq=0 out_seq=0 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.079+0000 7f6cf7fff640 10 client.15858 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6cf7fff640 1 -- 172.21.15.38:0/2408850416 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dec7bbfaa0 con 0x55dec7c63420 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6cf7fff640 10 client.15858 handle_command_reply: tid=0 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 client.15858 shutdown 2024-04-22T01:13:43.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 msgr2=0x55dec7c5ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 0x55dec7c5ea50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6d10003ba0 tx=0x7f6d10004a40 comp rx=0 tx=0).stop 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 shutdown_connections 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7c63420 0x55dec7c658e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c52ab0 0x55dec7c5ea50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c57a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6cf7fff640 0 client.15858 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 --2- 172.21.15.38:0/2408850416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c4cf50 0x55dec7c574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 >> 172.21.15.38:0/2408850416 conn(0x55dec7c484c0 msgr2=0x55dec7c476e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 shutdown_connections 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 1 -- 172.21.15.38:0/2408850416 wait complete. 2024-04-22T01:13:43.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.080+0000 7f6d21785740 20 client.15858 trim_cache size 0 max 16384 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 Processor -- start 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 -- start start 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4cf50 0x55dec7c46e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c5ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c47390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181ac790 con 0x55dec7c4d940 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181ae2d0 con 0x55dec7c52ab0 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d21785740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181ace60 con 0x55dec7c4cf50 2024-04-22T01:13:43.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c5ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c5ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53386/0 (socket says 172.21.15.38:53386) 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 -- 172.21.15.38:0/2174828144 learned_addr learned my addr 172.21.15.38:0/2174828144 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d1d137640 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c47390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 -- 172.21.15.38:0/2174828144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4cf50 msgr2=0x55dec7c46e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4cf50 0x55dec7c46e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 -- 172.21.15.38:0/2174828144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 msgr2=0x55dec7c47390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c47390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 -- 172.21.15.38:0/2174828144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec7c43fa0 con 0x55dec7c4d940 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d1d137640 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c47390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.100+0000 7f6d17fff640 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c5ea10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6d040eec70 tx=0x7f6d040f4c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e13558c5666149 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d027fc640 1 -- 172.21.15.38:0/2174828144 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0404b040 con 0x55dec7c4d940 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d027fc640 1 -- 172.21.15.38:0/2174828144 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d04005690 con 0x55dec7c4d940 2024-04-22T01:13:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d027fc640 1 -- 172.21.15.38:0/2174828144 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d040bdc50 con 0x55dec7c4d940 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 -- 172.21.15.38:0/2174828144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 msgr2=0x55dec7c5ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c5ea10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6d040eec70 tx=0x7f6d040f4c40 comp rx=0 tx=0).stop 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 -- 172.21.15.38:0/2174828144 shutdown_connections 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c47390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4d940 0x55dec7c5ea10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 --2- 172.21.15.38:0/2174828144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4cf50 0x55dec7c46e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 -- 172.21.15.38:0/2174828144 >> 172.21.15.38:0/2174828144 conn(0x55dec7c484c0 msgr2=0x55dec7c574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 -- 172.21.15.38:0/2174828144 shutdown_connections 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.101+0000 7f6d21785740 1 -- 172.21.15.38:0/2174828144 wait complete. 2024-04-22T01:13:43.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 -- start start 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 Processor -- start 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 0x55dec7c5c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c5c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c5d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181adc00 con 0x55dec7c4cf50 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181ad530 con 0x55dec7c52ab0 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181aa750 con 0x55dec7c4d940 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d21785740 10 client.0 _ll_register_callbacks cb 0x7f6d10009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d1c936640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 0x55dec7c5c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d1c936640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 0x55dec7c5c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53398/0 (socket says 172.21.15.38:53398) 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d1c936640 1 -- 172.21.15.38:0/286918394 learned_addr learned my addr 172.21.15.38:0/286918394 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.102+0000 7f6d17fff640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c5c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1d137640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c5d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1c936640 1 -- 172.21.15.38:0/286918394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 msgr2=0x55dec7c5c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1c936640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c5c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1c936640 1 -- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 msgr2=0x55dec7c5d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1c936640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c5d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1c936640 1 -- 172.21.15.38:0/286918394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec7c5dea0 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1d137640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c5d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d17fff640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c5c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d1c936640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 0x55dec7c5c360 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00bbd0 tx=0x7f6d0c005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be57066d0d74b496 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6cf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6cf7fff640 1 -- 172.21.15.38:0/286918394 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c00b770 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6cf7fff640 1 -- 172.21.15.38:0/286918394 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6d0c0285d0 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dec7bb9df0 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6cf7fff640 1 -- 172.21.15.38:0/286918394 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d0c034050 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d21785740 10 client.15864 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dec7cc9be0 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.103+0000 7f6cf7fff640 1 -- 172.21.15.38:0/286918394 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dec7bb9df0 con 0x55dec7c4cf50 2024-04-22T01:13:43.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6cf7fff640 1 -- 172.21.15.38:0/286918394 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6d0c0048d0 con 0x55dec7c4cf50 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d21785740 10 client.15864 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d21785740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d21785740 20 client.15864 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d21785740 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7cca2b0 0x55dec7ccc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d21785740 4 client.15864 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dec7b375a0 con 0x55dec7cca2b0 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d17fff640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7cca2b0 0x55dec7ccc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6cf7fff640 10 client.15864 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.104+0000 7f6d17fff640 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7cca2b0 0x55dec7ccc6d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42344b4de5a7b768 server_cookie=87cd0bf5ae666c4e in_seq=0 out_seq=0 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6cf7fff640 1 -- 172.21.15.38:0/286918394 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dec7b375a0 con 0x55dec7cca2b0 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6cf7fff640 10 client.15864 handle_command_reply: tid=0 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 client.15864 shutdown 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 msgr2=0x55dec7c5c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 0x55dec7c5c360 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00bbd0 tx=0x7f6d0c005900 comp rx=0 tx=0).stop 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 shutdown_connections 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55dec7cca2b0 0x55dec7ccc6d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dec7c52ab0 0x55dec7c5d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dec7c4d940 0x55dec7c5c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6cf7fff640 0 client.15864 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 --2- 172.21.15.38:0/286918394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dec7c4cf50 0x55dec7c5c360 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 >> 172.21.15.38:0/286918394 conn(0x55dec7c484c0 msgr2=0x55dec7c4b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 shutdown_connections 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 1 -- 172.21.15.38:0/286918394 wait complete. 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d21785740 20 client.15864 trim_cache size 0 max 16384 2024-04-22T01:13:43.111 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: "72afd7cb-73ab-4511-b7ef-e960ebcda3ae": { 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "72afd7cb-73ab-4511-b7ef-e960ebcda3ae", 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:13:43.112 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6d080a2ca0 msgr2=0x7f6d080a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6d080a2ca0 0x7f6d080a5160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55dec7c4e1a0 tx=0x7f6d10004ab0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 msgr2=0x7f6d18102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.105+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18102c50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6d0402ce70 tx=0x7f6d04005e40 comp rx=0 tx=0).stop 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 shutdown_connections 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6d080a2ca0 0x7f6d080a5160 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d18102230 0x7f6d18103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d18101870 0x7f6d18102c50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 --2- 172.21.15.38:0/4235243368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d18100680 0x7f6d18104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:43.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 >> 172.21.15.38:0/4235243368 conn(0x7f6d180fbbf0 msgr2=0x7f6d180fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:43.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 shutdown_connections 2024-04-22T01:13:43.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:13:43.106+0000 7f6d1eb9a640 1 -- 172.21.15.38:0/4235243368 wait complete. 2024-04-22T01:13:43.117 INFO:tasks.cephfs.filesystem:scrub status for tag:72afd7cb-73ab-4511-b7ef-e960ebcda3ae - {'path': '/', 'tag': '72afd7cb-73ab-4511-b7ef-e960ebcda3ae', 'options': 'recursive,force'} 2024-04-22T01:13:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:43 smithi111 ceph-mon[31057]: pgmap v396: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-22T01:13:43.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:43 smithi038 ceph-mon[28043]: pgmap v396: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-22T01:13:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:43 smithi203 ceph-mon[31825]: pgmap v396: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-22T01:13:44.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:44 smithi038 ceph-mon[28043]: scrub queued for path: / 2024-04-22T01:13:44.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:44 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:44.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:44 smithi038 ceph-mon[28043]: scrub summary: active paths [/] 2024-04-22T01:13:44.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:44 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:44 smithi203 ceph-mon[31825]: scrub queued for path: / 2024-04-22T01:13:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:44 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:44 smithi203 ceph-mon[31825]: scrub summary: active paths [/] 2024-04-22T01:13:44.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:44 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:44 smithi111 ceph-mon[31057]: scrub queued for path: / 2024-04-22T01:13:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:44 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:44 smithi111 ceph-mon[31057]: scrub summary: active paths [/] 2024-04-22T01:13:44.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:44 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:13:45.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:45 smithi038 ceph-mon[28043]: pgmap v397: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-22T01:13:45.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:45 smithi203 ceph-mon[31825]: pgmap v397: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-22T01:13:45.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:45 smithi111 ceph-mon[31057]: pgmap v397: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-22T01:13:47.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:47 smithi038 ceph-mon[28043]: pgmap v398: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T01:13:47.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:47 smithi038 ceph-mon[28043]: scrub summary: idle 2024-04-22T01:13:47.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:47 smithi203 ceph-mon[31825]: pgmap v398: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T01:13:47.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:47 smithi203 ceph-mon[31825]: scrub summary: idle 2024-04-22T01:13:47.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:47 smithi111 ceph-mon[31057]: pgmap v398: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T01:13:47.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:47 smithi111 ceph-mon[31057]: scrub summary: idle 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1649183758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 msgr2=0x7f2ce0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1649183758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce0100ac0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009900 tx=0x7f2cdc02bc30 comp rx=0 tx=0).stop 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1649183758 shutdown_connections 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1649183758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce0102250 0x7f2ce0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1649183758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce0101890 0x7f2ce0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1649183758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce0100ac0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1649183758 >> 172.21.15.38:0/1649183758 conn(0x7f2ce00fbc10 msgr2=0x7f2ce00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1649183758 shutdown_connections 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.513+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1649183758 wait complete. 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 Processor -- start 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 -- start start 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce01045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce0101890 0x7f2ce0102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce0102250 0x7f2ce01031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce010c320 con 0x7f2ce01006a0 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce010c1a0 con 0x7f2ce0101890 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce79c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce010c4a0 con 0x7f2ce0102250 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce01045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce01045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34178/0 (socket says 172.21.15.38:34178) 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 -- 172.21.15.38:0/1436977859 learned_addr learned my addr 172.21.15.38:0/1436977859 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce0102250 0x7f2ce01031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce0101890 0x7f2ce0102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 -- 172.21.15.38:0/1436977859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce0102250 msgr2=0x7f2ce01031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce0102250 0x7f2ce01031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 -- 172.21.15.38:0/1436977859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce0101890 msgr2=0x7f2ce0102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce0101890 0x7f2ce0102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.514+0000 7f2ce5765640 1 -- 172.21.15.38:0/1436977859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce0103a70 con 0x7f2ce01006a0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.515+0000 7f2ce5765640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce01045e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc005ab0 tx=0x7f2cdc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.515+0000 7f2cd67fc640 1 -- 172.21.15.38:0/1436977859 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc041070 con 0x7f2ce01006a0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.515+0000 7f2cd67fc640 1 -- 172.21.15.38:0/1436977859 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cdc0054e0 con 0x7f2ce01006a0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.515+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce01a1b50 con 0x7f2ce01006a0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.515+0000 7f2cd67fc640 1 -- 172.21.15.38:0/1436977859 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc034440 con 0x7f2ce01006a0 2024-04-22T01:13:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.515+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce01a2010 con 0x7f2ce01006a0 2024-04-22T01:13:49.519 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:49 smithi203 ceph-mon[31825]: pgmap v399: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T01:13:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.517+0000 7f2cd67fc640 1 -- 172.21.15.38:0/1436977859 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f2cdc044070 con 0x7f2ce01006a0 2024-04-22T01:13:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 Processor -- start 2024-04-22T01:13:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 -- start start 2024-04-22T01:13:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cd67fc640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2cd0 0x7f2ccc0a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f5d30 0x55c2e17f6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e17fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e17fdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c2e175ab70 con 0x55c2e17f6720 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c2e1703470 con 0x55c2e17f5d30 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c2e1756b30 con 0x55c2e17fb890 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2ce4f64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e17fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2cd0 0x7f2ccc0a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2cd67fc640 1 -- 172.21.15.38:0/1436977859 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f2cdc0f1000 con 0x7f2ce01006a0 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2ce4f64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e17fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34190/0 (socket says 172.21.15.38:34190) 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.518+0000 7f2ce4f64640 1 -- 172.21.15.38:0/3532594005 learned_addr learned my addr 172.21.15.38:0/3532594005 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e17fdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce5765640 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f5d30 0x55c2e17f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce4f64640 1 -- 172.21.15.38:0/3532594005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 msgr2=0x55c2e17fdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e17fdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce4f64640 1 -- 172.21.15.38:0/3532594005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f5d30 msgr2=0x55c2e17f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f5d30 0x55c2e17f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce4f64640 1 -- 172.21.15.38:0/3532594005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2e146f210 con 0x55c2e17f6720 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2cd0 0x7f2ccc0a5190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009fd0 tx=0x7f2cd8009ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e17fb220 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000b740 tx=0x7f2cd000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfaaf33881e4e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ccb7fe640 1 -- 172.21.15.38:0/3532594005 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0003dd0 con 0x55c2e17f6720 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ccb7fe640 1 -- 172.21.15.38:0/3532594005 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd0015070 con 0x55c2e17f6720 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.519+0000 7f2ccb7fe640 1 -- 172.21.15.38:0/3532594005 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0004e00 con 0x55c2e17f6720 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 -- 172.21.15.38:0/3532594005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 msgr2=0x55c2e17fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e17fb220 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000b740 tx=0x7f2cd000bc00 comp rx=0 tx=0).stop 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 -- 172.21.15.38:0/3532594005 shutdown_connections 2024-04-22T01:13:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e17fdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e17fb220 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/3532594005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f5d30 0x55c2e17f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 -- 172.21.15.38:0/3532594005 >> 172.21.15.38:0/3532594005 conn(0x55c2e17f12a0 msgr2=0x55c2e17f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 -- 172.21.15.38:0/3532594005 shutdown_connections 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.520+0000 7f2cea5b8740 1 -- 172.21.15.38:0/3532594005 wait complete. 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 -- start start 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 Processor -- start 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1800000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f6720 0x55c2e1800540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1807580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c2e17fe840 con 0x55c2e17f5d30 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c2e1716a10 con 0x55c2e17fb890 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c2e17fe9c0 con 0x55c2e17f6720 2024-04-22T01:13:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.521+0000 7f2cea5b8740 10 client.0 _ll_register_callbacks cb 0x55c2e1751b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1800000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1800000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34204/0 (socket says 172.21.15.38:34204) 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 -- 172.21.15.38:0/2605477575 learned_addr learned my addr 172.21.15.38:0/2605477575 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1807580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f6720 0x55c2e1800540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 -- 172.21.15.38:0/2605477575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f6720 msgr2=0x55c2e1800540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f6720 0x55c2e1800540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 -- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 msgr2=0x55c2e1807580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1807580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 -- 172.21.15.38:0/2605477575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2e153b620 con 0x55c2e17f5d30 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f6720 0x55c2e1800540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2ce5765640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1800000 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0bd1f0 tx=0x7f2cdc0b9190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c370dc613f68b000 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/2605477575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc04b040 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/2605477575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f2cdc003780 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cd0015070 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/2605477575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2cdc004c80 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cea5b8740 10 client.15876 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.522+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c2e180a8f0 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/2605477575 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd0015070 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/2605477575 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2cdc0c5c20 con 0x55c2e17f5d30 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cea5b8740 10 client.15876 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cea5b8740 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cea5b8740 20 client.15876 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e180c0c0 0x55c2e180e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cea5b8740 4 client.15876 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2e17680b0 con 0x55c2e180c0c0 2024-04-22T01:13:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.523+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e180c0c0 0x55c2e180e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.524+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e180c0c0 0x55c2e180e550 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a695c0b2c8ff193 server_cookie=f676198dad3adbfe in_seq=0 out_seq=0 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.524+0000 7f2cc8ff9640 10 client.15876 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.524+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/2605477575 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2e17680b0 con 0x55c2e180c0c0 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.524+0000 7f2cc8ff9640 10 client.15876 handle_command_reply: tid=0 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.524+0000 7f2cea5b8740 1 client.15876 shutdown 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 msgr2=0x55c2e1800000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1800000 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0bd1f0 tx=0x7f2cdc0b9190 comp rx=0 tx=0).stop 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 shutdown_connections 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e180c0c0 0x55c2e180e550 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1807580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f6720 0x55c2e1800540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cc8ff9640 0 client.15876 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2605477575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1800000 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 >> 172.21.15.38:0/2605477575 conn(0x55c2e17f12a0 msgr2=0x55c2e17f04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 shutdown_connections 2024-04-22T01:13:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2605477575 wait complete. 2024-04-22T01:13:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.525+0000 7f2cea5b8740 20 client.15876 trim_cache size 0 max 16384 2024-04-22T01:13:49.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 Processor -- start 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 -- start start 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f5d30 0x55c2e1807fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e1807540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1808510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01ac5e0 con 0x55c2e17f6720 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01ae120 con 0x55c2e17fb890 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01accb0 con 0x55c2e17f5d30 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2ce4f64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e1807540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2ce4f64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e1807540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34208/0 (socket says 172.21.15.38:34208) 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2ce4f64640 1 -- 172.21.15.38:0/2712256570 learned_addr learned my addr 172.21.15.38:0/2712256570 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1808510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.541+0000 7f2ce5765640 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f5d30 0x55c2e1807fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ce4f64640 1 -- 172.21.15.38:0/2712256570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f5d30 msgr2=0x55c2e1807fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f5d30 0x55c2e1807fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ce4f64640 1 -- 172.21.15.38:0/2712256570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 msgr2=0x55c2e1808510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1808510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ce4f64640 1 -- 172.21.15.38:0/2712256570 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2e17fac80 con 0x55c2e17f6720 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e1807540 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000b740 tx=0x7f2cd0004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14512880672fcd23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ccaffd640 1 -- 172.21.15.38:0/2712256570 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0003e50 con 0x55c2e17f6720 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ccaffd640 1 -- 172.21.15.38:0/2712256570 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd0015050 con 0x55c2e17f6720 2024-04-22T01:13:49.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.542+0000 7f2ccaffd640 1 -- 172.21.15.38:0/2712256570 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00117b0 con 0x55c2e17f6720 2024-04-22T01:13:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2712256570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 msgr2=0x55c2e1807540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e1807540 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000b740 tx=0x7f2cd0004700 comp rx=0 tx=0).stop 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2712256570 shutdown_connections 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17fb890 0x55c2e1808510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f6720 0x55c2e1807540 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/2712256570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17f5d30 0x55c2e1807fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2712256570 >> 172.21.15.38:0/2712256570 conn(0x55c2e17f12a0 msgr2=0x55c2e17f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2712256570 shutdown_connections 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 -- 172.21.15.38:0/2712256570 wait complete. 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 -- start start 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.543+0000 7f2cea5b8740 1 Processor -- start 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1804a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f6720 0x55c2e1804f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e1805f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01ada50 con 0x55c2e17f5d30 2024-04-22T01:13:49.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01ad380 con 0x55c2e17f6720 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01aa5a0 con 0x55c2e17fb890 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 10 client.0 _ll_register_callbacks cb 0x7f2cdc0b9860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1804a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1804a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34214/0 (socket says 172.21.15.38:34214) 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 -- 172.21.15.38:0/346192483 learned_addr learned my addr 172.21.15.38:0/346192483 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f6720 0x55c2e1804f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e1805f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 -- 172.21.15.38:0/346192483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 msgr2=0x55c2e1805f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e1805f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 -- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f6720 msgr2=0x55c2e1804f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f6720 0x55c2e1804f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 -- 172.21.15.38:0/346192483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2e1806540 con 0x55c2e17f5d30 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5f66640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e1805f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:13:49.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2ce5765640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1804a00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc02c710 tx=0x7f2cdc0bba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c958d30ca7c48bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/346192483 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0d5020 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/346192483 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f2cdc0ee630 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.544+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cd0015050 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/346192483 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2cdc0bd030 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 10 client.15882 fetch_fsmap learned FSMap version 28 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c2e18078c0 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/346192483 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd0015050 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/346192483 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2cdc0bc940 con 0x55c2e17f5d30 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 10 client.15882 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 20 client.15882 populate_metadata read hostname 'smithi038' 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e1872cc0 0x55c2e18750e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:13:49.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 4 client.15882 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:13:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c2e1762400 con 0x55c2e1872cc0 2024-04-22T01:13:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.545+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e1872cc0 0x55c2e18750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:13:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.546+0000 7f2cc8ff9640 10 client.15882 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.546+0000 7f2ce4f64640 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e1872cc0 0x55c2e18750e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5e5627b75d7c85 server_cookie=15f584dffbf760ed in_seq=0 out_seq=0 2024-04-22T01:13:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cc8ff9640 1 -- 172.21.15.38:0/346192483 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c2e1762400 con 0x55c2e1872cc0 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cc8ff9640 10 client.15882 handle_command_reply: tid=0 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 client.15882 shutdown 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 msgr2=0x55c2e1804a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1804a00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc02c710 tx=0x7f2cdc0bba00 comp rx=0 tx=0).stop 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 shutdown_connections 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55c2e1872cc0 0x55c2e18750e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c2e17fb890 0x55c2e1805f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2e17f6720 0x55c2e1804f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 --2- 172.21.15.38:0/346192483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2e17f5d30 0x55c2e1804a00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 >> 172.21.15.38:0/346192483 conn(0x55c2e17f12a0 msgr2=0x55c2e17f4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cc8ff9640 0 client.15882 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:13:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 shutdown_connections 2024-04-22T01:13:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 1 -- 172.21.15.38:0/346192483 wait complete. 2024-04-22T01:13:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.553+0000 7f2cea5b8740 20 client.15882 trim_cache size 0 max 16384 2024-04-22T01:13:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:13:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:13:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:13:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2cd0 msgr2=0x7f2ccc0a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2cd0 0x7f2ccc0a5190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009fd0 tx=0x7f2cd8009ee0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 msgr2=0x7f2ce01045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce01045e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc005ab0 tx=0x7f2cdc005dc0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 shutdown_connections 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f2ccc0a2cd0 0x7f2ccc0a5190 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ce0102250 0x7f2ce01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ce0101890 0x7f2ce0102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 --2- 172.21.15.38:0/1436977859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ce01006a0 0x7f2ce01045e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:13:49.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 >> 172.21.15.38:0/1436977859 conn(0x7f2ce00fbc10 msgr2=0x7f2ce00fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:13:49.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 shutdown_connections 2024-04-22T01:13:49.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:13:49.554+0000 7f2ce79c9640 1 -- 172.21.15.38:0/1436977859 wait complete. 2024-04-22T01:13:49.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:49 smithi111 ceph-mon[31057]: pgmap v399: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T01:13:49.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:49 smithi038 ceph-mon[28043]: pgmap v399: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T01:13:51.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:51 smithi203 ceph-mon[31825]: pgmap v400: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-04-22T01:13:51.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:51 smithi111 ceph-mon[31057]: pgmap v400: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-04-22T01:13:51.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:51 smithi038 ceph-mon[28043]: pgmap v400: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-04-22T01:13:53.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:53 smithi203 ceph-mon[31825]: pgmap v401: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:53.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:53 smithi111 ceph-mon[31057]: pgmap v401: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:53 smithi038 ceph-mon[28043]: pgmap v401: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-22T01:13:55.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:55 smithi203 ceph-mon[31825]: pgmap v402: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-22T01:13:55.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:55 smithi111 ceph-mon[31057]: pgmap v402: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-22T01:13:55.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:55 smithi038 ceph-mon[28043]: pgmap v402: 129 pgs: 129 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-22T01:13:57.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:57 smithi203 ceph-mon[31825]: pgmap v403: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-22T01:13:57.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:57 smithi111 ceph-mon[31057]: pgmap v403: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-22T01:13:57.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:57 smithi038 ceph-mon[28043]: pgmap v403: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 853 B/s wr, 12 op/s 2024-04-22T01:13:59.447 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:13:59 smithi111 ceph-mon[31057]: pgmap v404: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 853 B/s wr, 14 op/s 2024-04-22T01:13:59.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:13:59 smithi203 ceph-mon[31825]: pgmap v404: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 853 B/s wr, 14 op/s 2024-04-22T01:13:59.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:13:59 smithi038 ceph-mon[28043]: pgmap v404: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 853 B/s wr, 14 op/s 2024-04-22T01:14:01.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:01 smithi203 ceph-mon[31825]: pgmap v405: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T01:14:01.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:01 smithi111 ceph-mon[31057]: pgmap v405: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T01:14:01.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:01 smithi038 ceph-mon[28043]: pgmap v405: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T01:14:02.687 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 -- 172.21.15.38:0/1215494084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0102270 msgr2=0x7f1bc0108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 --2- 172.21.15.38:0/1215494084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0102270 0x7f1bc0108b00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009ca0 tx=0x7f1bb802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 -- 172.21.15.38:0/1215494084 shutdown_connections 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 --2- 172.21.15.38:0/1215494084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc0102270 0x7f1bc0108b00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 --2- 172.21.15.38:0/1215494084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc01018b0 0x7f1bc0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 --2- 172.21.15.38:0/1215494084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01006c0 0x7f1bc0100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 -- 172.21.15.38:0/1215494084 >> 172.21.15.38:0/1215494084 conn(0x7f1bc00fbc30 msgr2=0x7f1bc00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 -- 172.21.15.38:0/1215494084 shutdown_connections 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.686+0000 7f1bc7227640 1 -- 172.21.15.38:0/1215494084 wait complete. 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 Processor -- start 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 -- start start 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 0x7f1bc006aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01018b0 0x7f1bc019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc0102270 0x7f1bc019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010c340 con 0x7f1bc01006c0 2024-04-22T01:14:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010c1c0 con 0x7f1bc01018b0 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc7227640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010c4c0 con 0x7f1bc0102270 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc4fc3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 0x7f1bc006aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc4fc3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 0x7f1bc006aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52030/0 (socket says 172.21.15.38:52030) 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bc4fc3640 1 -- 172.21.15.38:0/882451627 learned_addr learned my addr 172.21.15.38:0/882451627 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.687+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01018b0 0x7f1bc019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc0102270 0x7f1bc019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc4fc3640 1 -- 172.21.15.38:0/882451627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc0102270 msgr2=0x7f1bc019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc0102270 0x7f1bc019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc4fc3640 1 -- 172.21.15.38:0/882451627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01018b0 msgr2=0x7f1bc019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01018b0 0x7f1bc019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc4fc3640 1 -- 172.21.15.38:0/882451627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc019f6b0 con 0x7f1bc01006c0 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc0102270 0x7f1bc019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01018b0 0x7f1bc019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:02.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 0x7f1bc006aa80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400e9c0 tx=0x7f1bb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bbdffb640 1 -- 172.21.15.38:0/882451627 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb400cd80 con 0x7f1bc01006c0 2024-04-22T01:14:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc019f940 con 0x7f1bc01006c0 2024-04-22T01:14:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bbdffb640 1 -- 172.21.15.38:0/882451627 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb40035f0 con 0x7f1bc01006c0 2024-04-22T01:14:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.688+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc019fee0 con 0x7f1bc01006c0 2024-04-22T01:14:02.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.689+0000 7f1bbdffb640 1 -- 172.21.15.38:0/882451627 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4010690 con 0x7f1bc01006c0 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 Processor -- start 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 -- start start 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 0x55739b022150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bbdffb640 1 -- 172.21.15.38:0/882451627 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f1bb40037a0 con 0x7f1bc01006c0 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b027220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b027890 0x55739b029d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55739af86b70 con 0x55739b022720 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55739af2f470 con 0x55739b027890 2024-04-22T01:14:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55739af82b30 con 0x55739b021d30 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc4fc3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b027220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc57c4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 0x55739b022150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc57c4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 0x55739b022150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:35572/0 (socket says 172.21.15.38:35572) 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc4fc3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b027220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52044/0 (socket says 172.21.15.38:52044) 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bc57c4640 1 -- 172.21.15.38:0/3172883613 learned_addr learned my addr 172.21.15.38:0/3172883613 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.692+0000 7f1bbffff640 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b027890 0x55739b029d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc57c4640 1 -- 172.21.15.38:0/3172883613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b027890 msgr2=0x55739b029d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b027890 0x55739b029d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc57c4640 1 -- 172.21.15.38:0/3172883613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 msgr2=0x55739b027220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bbdffb640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1bb00a2cd0 0x7f1bb00a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b027220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc57c4640 1 -- 172.21.15.38:0/3172883613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55739ac9b210 con 0x55739b021d30 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b027220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bbdffb640 1 -- 172.21.15.38:0/882451627 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f1bb40cfa80 con 0x7f1bc01006c0 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1bb00a2cd0 0x7f1bb00a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.693+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 0x55739b022150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8004af0 tx=0x7f1bb8005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2649d9b8dbee5a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.694+0000 7f1baaffd640 1 -- 172.21.15.38:0/3172883613 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8039050 con 0x55739b021d30 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.694+0000 7f1baaffd640 1 -- 172.21.15.38:0/3172883613 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb8045070 con 0x55739b021d30 2024-04-22T01:14:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.694+0000 7f1baaffd640 1 -- 172.21.15.38:0/3172883613 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8004280 con 0x55739b021d30 2024-04-22T01:14:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.694+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3172883613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 msgr2=0x55739b022150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.694+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1bb00a2cd0 0x7f1bb00a5190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1bc006a7f0 tx=0x7f1bac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.694+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 0x55739b022150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8004af0 tx=0x7f1bb8005f70 comp rx=0 tx=0).stop 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3172883613 shutdown_connections 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b027890 0x55739b029d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b027220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3172883613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b021d30 0x55739b022150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3172883613 >> 172.21.15.38:0/3172883613 conn(0x55739b01d2a0 msgr2=0x55739b01d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3172883613 shutdown_connections 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3172883613 wait complete. 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 -- start start 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.695+0000 7f1bc9e0f740 1 Processor -- start 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b021d30 0x55739b02bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 0x55739b02c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b033530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55739b02a7e0 con 0x55739b021d30 2024-04-22T01:14:02.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55739af42a10 con 0x55739b022720 2024-04-22T01:14:02.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55739b02a960 con 0x55739b027890 2024-04-22T01:14:02.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 0x55739b02c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc4fc3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b021d30 0x55739b02bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 0x55739b02c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:44518/0 (socket says 172.21.15.38:44518) 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 -- 172.21.15.38:0/3052412118 learned_addr learned my addr 172.21.15.38:0/3052412118 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b033530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 10 client.0 _ll_register_callbacks cb 0x55739af7db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 -- 172.21.15.38:0/3052412118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 msgr2=0x55739b033530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b033530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 -- 172.21.15.38:0/3052412118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b021d30 msgr2=0x55739b02bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b021d30 0x55739b02bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 -- 172.21.15.38:0/3052412118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55739ad67620 con 0x55739b022720 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b021d30 0x55739b02bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bb8045070 con 0x55739b022720 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1bbffff640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 0x55739b02c4f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1bac020a10 tx=0x7f1bac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfdd697f42ff8f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.696+0000 7f1b9ffff640 10 client.25439 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1b9ffff640 1 -- 172.21.15.38:0/3052412118 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac02d060 con 0x55739b022720 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1b9ffff640 1 -- 172.21.15.38:0/3052412118 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f1bac018030 con 0x55739b022720 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1b9ffff640 1 -- 172.21.15.38:0/3052412118 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1bac011940 con 0x55739b022720 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 10 client.25439 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55739b036840 con 0x55739b022720 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1b9ffff640 1 -- 172.21.15.38:0/3052412118 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1bac011ae0 con 0x55739b022720 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 10 client.25439 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 10 client.25439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 20 client.25439 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b038380 0x55739b03a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.697+0000 7f1bc9e0f740 4 client.25439 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.698+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55739af940b0 con 0x55739b038380 2024-04-22T01:14:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.698+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b038380 0x55739b03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.698+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b038380 0x55739b03a820 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=904cfd391bd7bd09 server_cookie=cf779e99aebfca2b in_seq=0 out_seq=0 2024-04-22T01:14:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.698+0000 7f1b9ffff640 10 client.25439 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1b9ffff640 1 -- 172.21.15.38:0/3052412118 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55739af940b0 con 0x55739b038380 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1b9ffff640 10 client.25439 handle_command_reply: tid=0 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 client.25439 shutdown 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 msgr2=0x55739b02c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 0x55739b02c4f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1bac020a10 tx=0x7f1bac004730 comp rx=0 tx=0).stop 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 shutdown_connections 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b038380 0x55739b03a820 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b033530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b022720 0x55739b02c4f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1b9ffff640 0 client.25439 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/3052412118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b021d30 0x55739b02bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 >> 172.21.15.38:0/3052412118 conn(0x55739b01d2a0 msgr2=0x55739b01c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 shutdown_connections 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/3052412118 wait complete. 2024-04-22T01:14:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.699+0000 7f1bc9e0f740 20 client.25439 trim_cache size 0 max 16384 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.714+0000 7f1bc9e0f740 1 Processor -- start 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 -- start start 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b03a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b022720 0x55739b0334f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 0x55739b018220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01a8200 con 0x55739b027890 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01a9d40 con 0x55739b021d30 2024-04-22T01:14:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01a88d0 con 0x55739b022720 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 0x55739b018220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 0x55739b018220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52054/0 (socket says 172.21.15.38:52054) 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 -- 172.21.15.38:0/1201160519 learned_addr learned my addr 172.21.15.38:0/1201160519 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bbffff640 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b022720 0x55739b0334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b03a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 -- 172.21.15.38:0/1201160519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b022720 msgr2=0x55739b0334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b022720 0x55739b0334f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 -- 172.21.15.38:0/1201160519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 msgr2=0x55739b03a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b03a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.715+0000 7f1bc57c4640 1 -- 172.21.15.38:0/1201160519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55739b01cbf0 con 0x55739b027890 2024-04-22T01:14:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.716+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 0x55739b018220 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x55739b022f80 tx=0x7f1bb802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da900528e1d0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.716+0000 7f1baa7fc640 1 -- 172.21.15.38:0/1201160519 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8032e90 con 0x55739b027890 2024-04-22T01:14:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.716+0000 7f1baa7fc640 1 -- 172.21.15.38:0/1201160519 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb803a040 con 0x55739b027890 2024-04-22T01:14:02.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.716+0000 7f1baa7fc640 1 -- 172.21.15.38:0/1201160519 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8004510 con 0x55739b027890 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/1201160519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 msgr2=0x55739b018220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 0x55739b018220 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x55739b022f80 tx=0x7f1bb802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/1201160519 shutdown_connections 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b027890 0x55739b018220 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b022720 0x55739b0334f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/1201160519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b03a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/1201160519 >> 172.21.15.38:0/1201160519 conn(0x55739b01d2a0 msgr2=0x55739b01d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/1201160519 shutdown_connections 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/1201160519 wait complete. 2024-04-22T01:14:02.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 -- start start 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.717+0000 7f1bc9e0f740 1 Processor -- start 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b030c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b0311b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b032100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01a9670 con 0x55739b022720 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01a8fa0 con 0x55739b021d30 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc01a61c0 con 0x55739b027890 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b0311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc9e0f740 10 client.0 _ll_register_callbacks cb 0x7f1bac009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b0311b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52068/0 (socket says 172.21.15.38:52068) 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 -- 172.21.15.38:0/882390533 learned_addr learned my addr 172.21.15.38:0/882390533 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b030c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bc57c4640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b032100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 -- 172.21.15.38:0/882390533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 msgr2=0x55739b032100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b032100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 -- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 msgr2=0x55739b030c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b030c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 -- 172.21.15.38:0/882390533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55739b0326f0 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.718+0000 7f1bbffff640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b0311b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1bac028630 tx=0x7f1bac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b845e3f207db8429 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1b9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1b9ffff640 1 -- 172.21.15.38:0/882390533 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac02d040 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bb803a040 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1b9ffff640 1 -- 172.21.15.38:0/882390533 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f1bac005450 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1b9ffff640 1 -- 172.21.15.38:0/882390533 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1bac044050 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 10 client.15900 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55739b09ea10 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1b9ffff640 1 -- 172.21.15.38:0/882390533 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb803a040 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1b9ffff640 1 -- 172.21.15.38:0/882390533 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1bac039020 con 0x55739b022720 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 10 client.15900 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 20 client.15900 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b09f380 0x55739b0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 4 client.15900 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.719+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55739af8e400 con 0x55739b09f380 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.720+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b09f380 0x55739b0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.720+0000 7f1b9ffff640 10 client.15900 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.720+0000 7f1bc4fc3640 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b09f380 0x55739b0a17a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8ceede24f8f7c53 server_cookie=2b08de7e7a6bbb82 in_seq=0 out_seq=0 2024-04-22T01:14:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1b9ffff640 1 -- 172.21.15.38:0/882390533 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55739af8e400 con 0x55739b09f380 2024-04-22T01:14:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1b9ffff640 10 client.15900 handle_command_reply: tid=0 2024-04-22T01:14:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 client.15900 shutdown 2024-04-22T01:14:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 msgr2=0x55739b0311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b0311b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1bac028630 tx=0x7f1bac019000 comp rx=0 tx=0).stop 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 shutdown_connections 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55739b09f380 0x55739b0a17a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55739b027890 0x55739b032100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55739b022720 0x55739b0311b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 --2- 172.21.15.38:0/882390533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55739b021d30 0x55739b030c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 >> 172.21.15.38:0/882390533 conn(0x55739b01d2a0 msgr2=0x55739b02bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1b9ffff640 0 client.15900 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 shutdown_connections 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 1 -- 172.21.15.38:0/882390533 wait complete. 2024-04-22T01:14:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.724+0000 7f1bc9e0f740 20 client.15900 trim_cache size 0 max 16384 2024-04-22T01:14:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:14:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:14:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:14:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1bb00a2cd0 msgr2=0x7f1bb00a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1bb00a2cd0 0x7f1bb00a5190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1bc006a7f0 tx=0x7f1bac005f90 comp rx=0 tx=0).stop 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 msgr2=0x7f1bc006aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 0x7f1bc006aa80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400e9c0 tx=0x7f1bb400ee80 comp rx=0 tx=0).stop 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 shutdown_connections 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f1bb00a2cd0 0x7f1bb00a5190 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bc0102270 0x7f1bc019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bc01018b0 0x7f1bc019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 --2- 172.21.15.38:0/882451627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bc01006c0 0x7f1bc006aa80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 >> 172.21.15.38:0/882451627 conn(0x7f1bc00fbc30 msgr2=0x7f1bc0104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 shutdown_connections 2024-04-22T01:14:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:02.725+0000 7f1bc7227640 1 -- 172.21.15.38:0/882451627 wait complete. 2024-04-22T01:14:03.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:03 smithi203 ceph-mon[31825]: pgmap v406: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-22T01:14:03.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:03 smithi111 ceph-mon[31057]: pgmap v406: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-22T01:14:03.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:03 smithi038 ceph-mon[28043]: pgmap v406: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-22T01:14:05.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:05 smithi203 ceph-mon[31825]: pgmap v407: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T01:14:05.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:05 smithi111 ceph-mon[31057]: pgmap v407: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T01:14:05.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:05 smithi038 ceph-mon[28043]: pgmap v407: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T01:14:07.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:07 smithi203 ceph-mon[31825]: pgmap v408: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T01:14:07.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:07 smithi111 ceph-mon[31057]: pgmap v408: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T01:14:07.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:07 smithi038 ceph-mon[28043]: pgmap v408: 129 pgs: 129 active+clean; 671 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T01:14:09.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:09 smithi203 ceph-mon[31825]: pgmap v409: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T01:14:09.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:09 smithi111 ceph-mon[31057]: pgmap v409: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T01:14:09.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:09 smithi038 ceph-mon[28043]: pgmap v409: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T01:14:10.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:10 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:14:10.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:10 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:14:10.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:10 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: pgmap v410: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:14:11.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: pgmap v410: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:14:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:14:11.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: pgmap v410: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:14:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:14:13.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:14:13.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:14:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 -- 172.21.15.38:0/139625648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c01090a0 msgr2=0x7fe9c010fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/139625648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c01090a0 0x7fe9c010fa80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009930 tx=0x7fe9b002bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 -- 172.21.15.38:0/139625648 shutdown_connections 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/139625648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9c010ffc0 0x7fe9c01123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/139625648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c01090a0 0x7fe9c010fa80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/139625648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c0108740 0x7fe9c0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 -- 172.21.15.38:0/139625648 >> 172.21.15.38:0/139625648 conn(0x7fe9c006e5d0 msgr2=0x7fe9c0070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 -- 172.21.15.38:0/139625648 shutdown_connections 2024-04-22T01:14:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.229+0000 7fe9c81d7640 1 -- 172.21.15.38:0/139625648 wait complete. 2024-04-22T01:14:13.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 Processor -- start 2024-04-22T01:14:13.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 -- start start 2024-04-22T01:14:13.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 0x7fe9c010b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c01090a0 0x7fe9c0109a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9c010ffc0 0x7fe9c0109f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c0115c30 con 0x7fe9c0108740 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c0115ab0 con 0x7fe9c01090a0 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c81d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c0115db0 con 0x7fe9c010ffc0 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c5f73640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 0x7fe9c010b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c5f73640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 0x7fe9c010b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48352/0 (socket says 172.21.15.38:48352) 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.230+0000 7fe9c5f73640 1 -- 172.21.15.38:0/2121982227 learned_addr learned my addr 172.21.15.38:0/2121982227 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c6774640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9c010ffc0 0x7fe9c0109f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5772640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c01090a0 0x7fe9c0109a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5f73640 1 -- 172.21.15.38:0/2121982227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9c010ffc0 msgr2=0x7fe9c0109f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9c010ffc0 0x7fe9c0109f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5f73640 1 -- 172.21.15.38:0/2121982227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c01090a0 msgr2=0x7fe9c0109a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c01090a0 0x7fe9c0109a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5f73640 1 -- 172.21.15.38:0/2121982227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c010a7e0 con 0x7fe9c0108740 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 0x7fe9c010b3d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00d950 tx=0x7fe9bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9b6ffd640 1 -- 172.21.15.38:0/2121982227 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc017030 con 0x7fe9c0108740 2024-04-22T01:14:13.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9b6ffd640 1 -- 172.21.15.38:0/2121982227 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9bc004020 con 0x7fe9c0108740 2024-04-22T01:14:13.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c01aa460 con 0x7fe9c0108740 2024-04-22T01:14:13.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9b6ffd640 1 -- 172.21.15.38:0/2121982227 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0047a0 con 0x7fe9c0108740 2024-04-22T01:14:13.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.231+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c01aa9d0 con 0x7fe9c0108740 2024-04-22T01:14:13.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.234+0000 7fe9b6ffd640 1 -- 172.21.15.38:0/2121982227 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7fe9bc0041d0 con 0x7fe9c0108740 2024-04-22T01:14:13.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.235+0000 7fe9b6ffd640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9ac0a2ca0 0x7fe9ac0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.235+0000 7fe9b6ffd640 1 -- 172.21.15.38:0/2121982227 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe9bc0cef40 con 0x7fe9c0108740 2024-04-22T01:14:13.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.235+0000 7fe9c5772640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9ac0a2ca0 0x7fe9ac0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.235+0000 7fe9cadc3740 1 Processor -- start 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.235+0000 7fe9cadc3740 1 -- start start 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9ac0a2ca0 0x7fe9ac0a5160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9b002c7b0 tx=0x7fe9b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f1a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f1f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f1fab0 0x55ef81f21f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f1f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f1f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48364/0 (socket says 172.21.15.38:48364) 2024-04-22T01:14:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 -- 172.21.15.38:0/894317181 learned_addr learned my addr 172.21.15.38:0/894317181 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ef81e7f560 con 0x55ef81f1a940 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ef81e27730 con 0x55ef81f1fab0 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ef81e7b520 con 0x55ef81f19f50 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f1a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c6774640 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f1fab0 0x55ef81f21f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 -- 172.21.15.38:0/894317181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 msgr2=0x55ef81f1a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f1a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 -- 172.21.15.38:0/894317181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f1fab0 msgr2=0x55ef81f21f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f1fab0 0x55ef81f21f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c5772640 1 -- 172.21.15.38:0/894317181 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef81c5d7d0 con 0x55ef81f1a940 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.236+0000 7fe9c6774640 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f1fab0 0x55ef81f21f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.237+0000 7fe9c5772640 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f1f440 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0005ec0 tx=0x7fe9b003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81c466230338a04 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.237+0000 7fe9abfff640 1 -- 172.21.15.38:0/894317181 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b004c030 con 0x55ef81f1a940 2024-04-22T01:14:13.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.237+0000 7fe9abfff640 1 -- 172.21.15.38:0/894317181 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b0005530 con 0x55ef81f1a940 2024-04-22T01:14:13.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.237+0000 7fe9abfff640 1 -- 172.21.15.38:0/894317181 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b00035f0 con 0x55ef81f1a940 2024-04-22T01:14:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 msgr2=0x55ef81f1f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f1f440 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0005ec0 tx=0x7fe9b003e000 comp rx=0 tx=0).stop 2024-04-22T01:14:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 shutdown_connections 2024-04-22T01:14:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f1fab0 0x55ef81f21f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f1f440 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/894317181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f1a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 >> 172.21.15.38:0/894317181 conn(0x55ef81f154c0 msgr2=0x55ef81f158c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 shutdown_connections 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 -- 172.21.15.38:0/894317181 wait complete. 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 -- start start 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.238+0000 7fe9cadc3740 1 Processor -- start 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f24a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f1fab0 0x55ef81f2ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ef81f22a00 con 0x55ef81f1a940 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ef81e3ae00 con 0x55ef81f19f50 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ef81f22b80 con 0x55ef81f1fab0 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9c5772640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f24a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9c5772640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f24a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48368/0 (socket says 172.21.15.38:48368) 2024-04-22T01:14:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9c5772640 1 -- 172.21.15.38:0/4066498549 learned_addr learned my addr 172.21.15.38:0/4066498549 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9cadc3740 10 client.0 _ll_register_callbacks cb 0x55ef81e75eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9c6774640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f1fab0 0x55ef81f2ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.239+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5772640 1 -- 172.21.15.38:0/4066498549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f1fab0 msgr2=0x55ef81f2ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5772640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f1fab0 0x55ef81f2ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5772640 1 -- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 msgr2=0x55ef81f244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5772640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f244c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5772640 1 -- 172.21.15.38:0/4066498549 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef81b933f0 con 0x55ef81f1a940 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c6774640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f1fab0 0x55ef81f2ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9c5772640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f24a00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0036040 tx=0x7fe9b0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec1547c721ed4bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9a97fa640 1 -- 172.21.15.38:0/4066498549 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b004c040 con 0x55ef81f1a940 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9a97fa640 1 -- 172.21.15.38:0/4066498549 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe9b0005250 con 0x55ef81f1a940 2024-04-22T01:14:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.240+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b0005530 con 0x55ef81f1a940 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9a97fa640 1 -- 172.21.15.38:0/4066498549 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe9b00035f0 con 0x55ef81f1a940 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 10 client.15912 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef81b530f0 con 0x55ef81f1a940 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9a97fa640 1 -- 172.21.15.38:0/4066498549 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b0005530 con 0x55ef81f1a940 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9a97fa640 1 -- 172.21.15.38:0/4066498549 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe9b00488b0 con 0x55ef81f1a940 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 10 client.15912 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 10 client.15912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 20 client.15912 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f2ff30 0x55ef81f32320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 4 client.15912 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.241+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef81e8caa0 con 0x55ef81f2ff30 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.242+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f2ff30 0x55ef81f32320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.242+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f2ff30 0x55ef81f32320 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=250221264994b03e server_cookie=2dab2bbccadd0dc7 in_seq=0 out_seq=0 2024-04-22T01:14:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.242+0000 7fe9a97fa640 10 client.15912 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9a97fa640 1 -- 172.21.15.38:0/4066498549 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef81e8caa0 con 0x55ef81f2ff30 2024-04-22T01:14:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9a97fa640 10 client.15912 handle_command_reply: tid=0 2024-04-22T01:14:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 client.15912 shutdown 2024-04-22T01:14:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 msgr2=0x55ef81f24a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f24a00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0036040 tx=0x7fe9b0004740 comp rx=0 tx=0).stop 2024-04-22T01:14:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 shutdown_connections 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f2ff30 0x55ef81f32320 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9a97fa640 0 client.15912 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f1fab0 0x55ef81f2ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f24a00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/4066498549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.243+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 >> 172.21.15.38:0/4066498549 conn(0x55ef81f154c0 msgr2=0x55ef81f146e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.244+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 shutdown_connections 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.244+0000 7fe9cadc3740 1 -- 172.21.15.38:0/4066498549 wait complete. 2024-04-22T01:14:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.244+0000 7fe9cadc3740 20 client.15912 trim_cache size 0 max 16384 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 Processor -- start 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 -- start start 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f154c0 0x55ef81f14e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f2ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f18e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b4fb0 con 0x55ef81f1a940 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b6af0 con 0x55ef81f154c0 2024-04-22T01:14:13.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b5680 con 0x55ef81f19f50 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9c6774640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f18e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9c6774640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f18e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48372/0 (socket says 172.21.15.38:48372) 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9c6774640 1 -- 172.21.15.38:0/2792499321 learned_addr learned my addr 172.21.15.38:0/2792499321 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9c5772640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f2ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f154c0 0x55ef81f14e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.262+0000 7fe9c6774640 1 -- 172.21.15.38:0/2792499321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 msgr2=0x55ef81f2ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c6774640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f2ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c6774640 1 -- 172.21.15.38:0/2792499321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f154c0 msgr2=0x55ef81f14e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c6774640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f154c0 0x55ef81f14e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c6774640 1 -- 172.21.15.38:0/2792499321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef81f16230 con 0x55ef81f1a940 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f154c0 0x55ef81f14e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c5772640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f2ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9c6774640 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f18e30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009f90 tx=0x7fe9b8012880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322086a3a24d4c7a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9ab7fe640 1 -- 172.21.15.38:0/2792499321 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8003a30 con 0x55ef81f1a940 2024-04-22T01:14:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9ab7fe640 1 -- 172.21.15.38:0/2792499321 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b8003bd0 con 0x55ef81f1a940 2024-04-22T01:14:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.263+0000 7fe9ab7fe640 1 -- 172.21.15.38:0/2792499321 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8010890 con 0x55ef81f1a940 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 -- 172.21.15.38:0/2792499321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 msgr2=0x55ef81f18e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f18e30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009f90 tx=0x7fe9b8012880 comp rx=0 tx=0).stop 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 -- 172.21.15.38:0/2792499321 shutdown_connections 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f18e30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f19f50 0x55ef81f2ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/2792499321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f154c0 0x55ef81f14e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 -- 172.21.15.38:0/2792499321 >> 172.21.15.38:0/2792499321 conn(0x55ef81f32690 msgr2=0x55ef81f18720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 -- 172.21.15.38:0/2792499321 shutdown_connections 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.264+0000 7fe9cadc3740 1 -- 172.21.15.38:0/2792499321 wait complete. 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 -- start start 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 Processor -- start 2024-04-22T01:14:13.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f154c0 0x55ef81f2b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f2b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f297f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b6420 con 0x55ef81f1a940 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9cadc3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b5d50 con 0x55ef81f19f50 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9c6774640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f297f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.265+0000 7fe9c6774640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f297f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48388/0 (socket says 172.21.15.38:48388) 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 -- 172.21.15.38:0/1326207086 learned_addr learned my addr 172.21.15.38:0/1326207086 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b2f70 con 0x55ef81f154c0 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9cadc3740 10 client.0 _ll_register_callbacks cb 0x7fe9bc0051d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c5772640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f2b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 -- 172.21.15.38:0/1326207086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f154c0 msgr2=0x55ef81f2b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f154c0 0x55ef81f2b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 -- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 msgr2=0x55ef81f2b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f2b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 -- 172.21.15.38:0/1326207086 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef81f29de0 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c5772640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f2b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9c6774640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f297f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8004280 tx=0x7fe9b80042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6122b75cdd1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9a97fa640 1 -- 172.21.15.38:0/1326207086 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8021040 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9a97fa640 1 -- 172.21.15.38:0/1326207086 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe9b8003d80 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.266+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b8003bd0 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.267+0000 7fe9a97fa640 1 -- 172.21.15.38:0/1326207086 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe9b8033050 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.267+0000 7fe9cadc3740 10 client.15918 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.267+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef81f96c80 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.267+0000 7fe9a97fa640 1 -- 172.21.15.38:0/1326207086 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b802a070 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9a97fa640 1 -- 172.21.15.38:0/1326207086 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe9b8024070 con 0x55ef81f1a940 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 10 client.15918 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:13.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 10 client.15918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 20 client.15918 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f1fab0 0x55ef81f99640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 4 client.15918 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe9b8003bd0 con 0x55ef81f1fab0 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f1fab0 0x55ef81f99640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9c5f73640 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f1fab0 0x55ef81f99640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f6384935fa3d01 server_cookie=7ce3b4b51f4ce551 in_seq=0 out_seq=0 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9a97fa640 10 client.15918 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9a97fa640 1 -- 172.21.15.38:0/1326207086 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe9b8003bd0 con 0x55ef81f1fab0 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9a97fa640 10 client.15918 handle_command_reply: tid=0 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.268+0000 7fe9cadc3740 1 client.15918 shutdown 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 msgr2=0x55ef81f297f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f297f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8004280 tx=0x7fe9b80042b0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 shutdown_connections 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ef81f1fab0 0x55ef81f99640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef81f1a940 0x55ef81f297f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9a97fa640 0 client.15918 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef81f19f50 0x55ef81f2b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 --2- 172.21.15.38:0/1326207086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef81f154c0 0x55ef81f2b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 >> 172.21.15.38:0/1326207086 conn(0x55ef81f32690 msgr2=0x55ef81f18fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 shutdown_connections 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 1 -- 172.21.15.38:0/1326207086 wait complete. 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.269+0000 7fe9cadc3740 20 client.15918 trim_cache size 0 max 16384 2024-04-22T01:14:13.273 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:14:13.274 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T01:14:13.274 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T01:14:13.274 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9ac0a2ca0 msgr2=0x7fe9ac0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9ac0a2ca0 0x7fe9ac0a5160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9b002c7b0 tx=0x7fe9b0005fd0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 msgr2=0x7fe9c010b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 0x7fe9c010b3d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00d950 tx=0x7fe9bc00de10 comp rx=0 tx=0).stop 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 shutdown_connections 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe9ac0a2ca0 0x7fe9ac0a5160 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9c010ffc0 0x7fe9c0109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9c01090a0 0x7fe9c0109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 --2- 172.21.15.38:0/2121982227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9c0108740 0x7fe9c010b3d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 >> 172.21.15.38:0/2121982227 conn(0x7fe9c006e5d0 msgr2=0x7fe9c006eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 shutdown_connections 2024-04-22T01:14:13.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.270+0000 7fe9c81d7640 1 -- 172.21.15.38:0/2121982227 wait complete. 2024-04-22T01:14:13.281 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T01:14:13.281 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:14:13.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 -- 172.21.15.38:0/73280589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30107460 msgr2=0x7f0b301098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 --2- 172.21.15.38:0/73280589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30107460 0x7f0b301098b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009ca0 tx=0x7f0b2802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 -- 172.21.15.38:0/73280589 shutdown_connections 2024-04-22T01:14:13.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 --2- 172.21.15.38:0/73280589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30107460 0x7f0b301098b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 --2- 172.21.15.38:0/73280589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b30104b30 0x7f0b30106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 --2- 172.21.15.38:0/73280589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b30102200 0x7f0b301045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 -- 172.21.15.38:0/73280589 >> 172.21.15.38:0/73280589 conn(0x7f0b300fbbf0 msgr2=0x7f0b300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 -- 172.21.15.38:0/73280589 shutdown_connections 2024-04-22T01:14:13.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.387+0000 7f0b349d7640 1 -- 172.21.15.38:0/73280589 wait complete. 2024-04-22T01:14:13.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 Processor -- start 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 -- start start 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 0x7f0b30101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b30104b30 0x7f0b301001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b30107460 0x7f0b301006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010d420 con 0x7f0b30102200 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010d2a0 con 0x7f0b30104b30 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b349d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010d5a0 con 0x7f0b30107460 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 0x7f0b30101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 0x7f0b30101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48406/0 (socket says 172.21.15.38:48406) 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 -- 172.21.15.38:0/3282117104 learned_addr learned my addr 172.21.15.38:0/3282117104 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2dd9b640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b30104b30 0x7f0b301001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 -- 172.21.15.38:0/3282117104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b30107460 msgr2=0x7f0b301006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b30107460 0x7f0b301006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 -- 172.21.15.38:0/3282117104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b30104b30 msgr2=0x7f0b301001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b30104b30 0x7f0b301001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.388+0000 7f0b2e59c640 1 -- 172.21.15.38:0/3282117104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b30100f80 con 0x7f0b30102200 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.389+0000 7f0b2e59c640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 0x7f0b30101b70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000d950 tx=0x7f0b2000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.389+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20017030 con 0x7f0b30102200 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.389+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b301a1b60 con 0x7f0b30102200 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.389+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b20004020 con 0x7f0b30102200 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.389+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b200047a0 con 0x7f0b30102200 2024-04-22T01:14:13.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.389+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b301a2100 con 0x7f0b30102200 2024-04-22T01:14:13.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.390+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b300fd6e0 con 0x7f0b30102200 2024-04-22T01:14:13.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.391+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f0b200041d0 con 0x7f0b30102200 2024-04-22T01:14:13.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.392+0000 7f0b1f7fe640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0b180a2ca0 0x7f0b180a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.392+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f0b200cf7e0 con 0x7f0b30102200 2024-04-22T01:14:13.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.392+0000 7f0b2dd9b640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0b180a2ca0 0x7f0b180a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.393+0000 7f0b2dd9b640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0b180a2ca0 0x7f0b180a5160 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005fd0 tx=0x7f0b2400b510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.400+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0b2009c1d0 con 0x7f0b30102200 2024-04-22T01:14:13.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.629+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b3002d360 con 0x7f0b30102200 2024-04-22T01:14:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.630+0000 7f0b1f7fe640 1 -- 172.21.15.38:0/3282117104 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f0b2009bad0 con 0x7f0b30102200 2024-04-22T01:14:13.629 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:14:13.629 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:14:13.630 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0b180a2ca0 msgr2=0x7f0b180a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0b180a2ca0 0x7f0b180a5160 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005fd0 tx=0x7f0b2400b510 comp rx=0 tx=0).stop 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 msgr2=0x7f0b30101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 0x7f0b30101b70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000d950 tx=0x7f0b2000de10 comp rx=0 tx=0).stop 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 shutdown_connections 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0b180a2ca0 0x7f0b180a5160 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b30107460 0x7f0b301006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b30104b30 0x7f0b301001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 --2- 172.21.15.38:0/3282117104 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b30102200 0x7f0b30101b70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 >> 172.21.15.38:0/3282117104 conn(0x7f0b300fbbf0 msgr2=0x7f0b30102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 shutdown_connections 2024-04-22T01:14:13.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.633+0000 7f0b349d7640 1 -- 172.21.15.38:0/3282117104 wait complete. 2024-04-22T01:14:13.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 -- 172.21.15.38:0/816591249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc102200 msgr2=0x7f53fc1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 --2- 172.21.15.38:0/816591249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc102200 0x7f53fc1045f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009900 tx=0x7f53f802bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 -- 172.21.15.38:0/816591249 shutdown_connections 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 --2- 172.21.15.38:0/816591249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53fc107460 0x7f53fc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 --2- 172.21.15.38:0/816591249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53fc104b30 0x7f53fc106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 --2- 172.21.15.38:0/816591249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc102200 0x7f53fc1045f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 -- 172.21.15.38:0/816591249 >> 172.21.15.38:0/816591249 conn(0x7f53fc0fbbf0 msgr2=0x7f53fc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 -- 172.21.15.38:0/816591249 shutdown_connections 2024-04-22T01:14:13.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.758+0000 7f540473e640 1 -- 172.21.15.38:0/816591249 wait complete. 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 Processor -- start 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 -- start start 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53fc102200 0x7f53fc199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53fc104b30 0x7f53fc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 0x7f53fc1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc10d420 con 0x7f53fc107460 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc10d2a0 con 0x7f53fc102200 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f540473e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc10d5a0 con 0x7f53fc104b30 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f5402cdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 0x7f53fc1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f5402cdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 0x7f53fc1a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48422/0 (socket says 172.21.15.38:48422) 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f5402cdb640 1 -- 172.21.15.38:0/480639257 learned_addr learned my addr 172.21.15.38:0/480639257 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f5401cd9640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53fc104b30 0x7f53fc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.759+0000 7f54024da640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53fc102200 0x7f53fc199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f5402cdb640 1 -- 172.21.15.38:0/480639257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53fc104b30 msgr2=0x7f53fc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f5402cdb640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53fc104b30 0x7f53fc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f5402cdb640 1 -- 172.21.15.38:0/480639257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53fc102200 msgr2=0x7f53fc199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f5402cdb640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53fc102200 0x7f53fc199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f5402cdb640 1 -- 172.21.15.38:0/480639257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53fc1a1960 con 0x7f53fc107460 2024-04-22T01:14:13.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f5402cdb640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 0x7f53fc1a12c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f53f400eaa0 tx=0x7f53f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f53f37fe640 1 -- 172.21.15.38:0/480639257 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f400cdb0 con 0x7f53fc107460 2024-04-22T01:14:13.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f53f37fe640 1 -- 172.21.15.38:0/480639257 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53f4005020 con 0x7f53fc107460 2024-04-22T01:14:13.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53fc1a1c50 con 0x7f53fc107460 2024-04-22T01:14:13.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f53f37fe640 1 -- 172.21.15.38:0/480639257 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f40116b0 con 0x7f53fc107460 2024-04-22T01:14:13.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.760+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53fc1a21a0 con 0x7f53fc107460 2024-04-22T01:14:13.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.763+0000 7f5407408740 1 Processor -- start 2024-04-22T01:14:13.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.763+0000 7f53f37fe640 1 -- 172.21.15.38:0/480639257 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f53f4011850 con 0x7f53fc107460 2024-04-22T01:14:13.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 -- start start 2024-04-22T01:14:13.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 0x56167c8ddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8de240 0x56167c8e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8e33b0 0x56167c8e5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56167c83a530 con 0x56167c8dd850 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56167c7e2830 con 0x56167c8e33b0 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5407408740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56167c8364f0 con 0x56167c8de240 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5401cd9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 0x56167c8ddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f54024da640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8de240 0x56167c8e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5401cd9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 0x56167c8ddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48432/0 (socket says 172.21.15.38:48432) 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5401cd9640 1 -- 172.21.15.38:0/4057161284 learned_addr learned my addr 172.21.15.38:0/4057161284 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.764+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8e33b0 0x56167c8e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5401cd9640 1 -- 172.21.15.38:0/4057161284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8de240 msgr2=0x56167c8e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8de240 0x56167c8e2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5401cd9640 1 -- 172.21.15.38:0/4057161284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8e33b0 msgr2=0x56167c8e5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f53f37fe640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53e80a2d70 0x7f53e80a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8e33b0 0x56167c8e5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5401cd9640 1 -- 172.21.15.38:0/4057161284 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167c8062c0 con 0x56167c8dd850 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 0x56167c8ddc70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f53ec011680 tx=0x7f53ec009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb386cf27e13696c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f53f37fe640 1 -- 172.21.15.38:0/480639257 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f53f40cf6d0 con 0x7f53fc107460 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f5402cdb640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53e80a2d70 0x7f53e80a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f53f0ff9640 1 -- 172.21.15.38:0/4057161284 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec01a030 con 0x56167c8dd850 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f53f0ff9640 1 -- 172.21.15.38:0/4057161284 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec00fa60 con 0x56167c8dd850 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.765+0000 7f53f0ff9640 1 -- 172.21.15.38:0/4057161284 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec01eb40 con 0x56167c8dd850 2024-04-22T01:14:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 -- 172.21.15.38:0/4057161284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 msgr2=0x56167c8ddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5402cdb640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53e80a2d70 0x7f53e80a5230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009270 tx=0x7f53f4097cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 0x56167c8ddc70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f53ec011680 tx=0x7f53ec009950 comp rx=0 tx=0).stop 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 -- 172.21.15.38:0/4057161284 shutdown_connections 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8e33b0 0x56167c8e5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8de240 0x56167c8e2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 --2- 172.21.15.38:0/4057161284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8dd850 0x56167c8ddc70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 -- 172.21.15.38:0/4057161284 >> 172.21.15.38:0/4057161284 conn(0x56167c8d8dc0 msgr2=0x56167c8d91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 -- 172.21.15.38:0/4057161284 shutdown_connections 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.766+0000 7f5407408740 1 -- 172.21.15.38:0/4057161284 wait complete. 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 -- start start 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 Processor -- start 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8eef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56167c8e6360 con 0x56167c8e33b0 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56167c7f5f10 con 0x56167c8de240 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56167c8e64e0 con 0x56167c8dd850 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5407408740 10 client.0 _ll_register_callbacks cb 0x56167c54dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8eef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48438/0 (socket says 172.21.15.38:48438) 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 -- 172.21.15.38:0/1846739043 learned_addr learned my addr 172.21.15.38:0/1846739043 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f54024da640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 -- 172.21.15.38:0/1846739043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 msgr2=0x56167c8e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 -- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 msgr2=0x56167c8e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.767+0000 7f5401cd9640 1 -- 172.21.15.38:0/1846739043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167c7c7c80 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f5401cd9640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8eef10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f53ec01e610 tx=0x7f53ec0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b21d98aaa5a763 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f53e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f53e5ffb640 1 -- 172.21.15.38:0/1846739043 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec01a050 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f53e5ffb640 1 -- 172.21.15.38:0/1846739043 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f53ec004220 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53ec00fa60 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f53e5ffb640 1 -- 172.21.15.38:0/1846739043 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53ec004420 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f5407408740 10 client.15936 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.768+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56167c54e3f0 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f53e5ffb640 1 -- 172.21.15.38:0/1846739043 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec00fa60 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f53e5ffb640 1 -- 172.21.15.38:0/1846739043 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53ec003a00 con 0x56167c8e33b0 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f5407408740 10 client.15936 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f5407408740 10 client.15936 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f5407408740 20 client.15936 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f5407408740 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c8f3590 0x56167c8f5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f5407408740 4 client.15936 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:13.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56167c847a70 con 0x56167c8f3590 2024-04-22T01:14:13.772 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:13 smithi203 ceph-mon[31825]: pgmap v411: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:14:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.769+0000 7f54024da640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c8f3590 0x56167c8f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f53e5ffb640 10 client.15936 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f54024da640 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c8f3590 0x56167c8f5980 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2b3dfea0b9386b8c server_cookie=b72f5a3a2edf928 in_seq=0 out_seq=0 2024-04-22T01:14:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f53e5ffb640 1 -- 172.21.15.38:0/1846739043 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56167c847a70 con 0x56167c8f3590 2024-04-22T01:14:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f53e5ffb640 10 client.15936 handle_command_reply: tid=0 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f5407408740 1 client.15936 shutdown 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 msgr2=0x56167c8eef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.770+0000 7f5407408740 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8eef10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f53ec01e610 tx=0x7f53ec0056d0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 shutdown_connections 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c8f3590 0x56167c8f5980 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8eef10 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 --2- 172.21.15.38:0/1846739043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 >> 172.21.15.38:0/1846739043 conn(0x56167c8d8dc0 msgr2=0x56167c8cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f53e5ffb640 0 client.15936 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 shutdown_connections 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 1 -- 172.21.15.38:0/1846739043 wait complete. 2024-04-22T01:14:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.771+0000 7f5407408740 20 client.15936 trim_cache size 0 max 16384 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.797+0000 7f5407408740 1 Processor -- start 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.797+0000 7f5407408740 1 -- start start 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5407408740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5407408740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8eee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5407408740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8fa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5407408740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1a8350 con 0x56167c8e33b0 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5407408740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1a9e90 con 0x56167c8de240 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5407408740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1a8a20 con 0x56167c8dd850 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8fa240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48442/0 (socket says 172.21.15.38:48442) 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 -- 172.21.15.38:0/2710301519 learned_addr learned my addr 172.21.15.38:0/2710301519 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f54024da640 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 -- 172.21.15.38:0/2710301519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 msgr2=0x56167c8cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 -- 172.21.15.38:0/2710301519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 msgr2=0x56167c8eee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8eee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 -- 172.21.15.38:0/2710301519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167c6187d0 con 0x56167c8e33b0 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f54024da640 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.798+0000 7f5401cd9640 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8fa240 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009d40 tx=0x7f53ec005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c467d0b965d45a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f53e7fff640 1 -- 172.21.15.38:0/2710301519 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec01a040 con 0x56167c8e33b0 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f53e7fff640 1 -- 172.21.15.38:0/2710301519 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec01b040 con 0x56167c8e33b0 2024-04-22T01:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f53e7fff640 1 -- 172.21.15.38:0/2710301519 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec003fc0 con 0x56167c8e33b0 2024-04-22T01:14:13.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 -- 172.21.15.38:0/2710301519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 msgr2=0x56167c8fa240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8fa240 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009d40 tx=0x7f53ec005970 comp rx=0 tx=0).stop 2024-04-22T01:14:13.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 -- 172.21.15.38:0/2710301519 shutdown_connections 2024-04-22T01:14:13.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8e33b0 0x56167c8fa240 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8de240 0x56167c8eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 --2- 172.21.15.38:0/2710301519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8dd850 0x56167c8cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 -- 172.21.15.38:0/2710301519 >> 172.21.15.38:0/2710301519 conn(0x56167c8d8dc0 msgr2=0x56167c8cefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.799+0000 7f5407408740 1 -- 172.21.15.38:0/2710301519 shutdown_connections 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f5407408740 1 -- 172.21.15.38:0/2710301519 wait complete. 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f5407408740 1 -- start start 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f5407408740 1 Processor -- start 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f5407408740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8dd850 0x56167c8ecbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f5407408740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 0x56167c8ed110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f54024da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 0x56167c8ed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f54024da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 0x56167c8ed110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48444/0 (socket says 172.21.15.38:48444) 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f54024da640 1 -- 172.21.15.38:0/4147238593 learned_addr learned my addr 172.21.15.38:0/4147238593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.800+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8ee0d0 0x56167c8ee550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1a97c0 con 0x56167c8de240 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1a90f0 con 0x56167c8dd850 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1a6310 con 0x56167c8ee0d0 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8dd850 0x56167c8ecbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8ee0d0 0x56167c8ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f54024da640 1 -- 172.21.15.38:0/4147238593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8ee0d0 msgr2=0x56167c8ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f54024da640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8ee0d0 0x56167c8ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f54024da640 1 -- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8dd850 msgr2=0x56167c8ecbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f54024da640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8dd850 0x56167c8ecbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f54024da640 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167c8eea90 con 0x56167c8de240 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8dd850 0x56167c8ecbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5407408740 10 client.0 _ll_register_callbacks cb 0x7f53f8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53ec01b040 con 0x56167c8de240 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f54024da640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 0x56167c8ed110 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f53f802c7b0 tx=0x7f53f802cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba48d915a027cea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.801+0000 7f53e5ffb640 10 client.15942 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f53e5ffb640 1 -- 172.21.15.38:0/4147238593 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8054050 con 0x56167c8de240 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f53e5ffb640 1 -- 172.21.15.38:0/4147238593 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f53f8004450 con 0x56167c8de240 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f53e5ffb640 1 -- 172.21.15.38:0/4147238593 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53f804a070 con 0x56167c8de240 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 10 client.15942 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=28}) -- 0x56167c95a2b0 con 0x56167c8de240 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f53e5ffb640 1 -- 172.21.15.38:0/4147238593 <== mon.0 v2:172.21.15.38:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53f80463a0 con 0x56167c8de240 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 10 client.15942 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 10 client.15942 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 20 client.15942 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c95ab50 0x56167c95cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.802+0000 7f5407408740 4 client.15942 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56167c841dc0 con 0x56167c95ab50 2024-04-22T01:14:13.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c95ab50 0x56167c95cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f5401cd9640 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c95ab50 0x56167c95cf70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=30521aa9fb616801 server_cookie=d25f0e0973ac5386 in_seq=0 out_seq=0 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f53e5ffb640 10 client.15942 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f53e5ffb640 1 -- 172.21.15.38:0/4147238593 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56167c841dc0 con 0x56167c95ab50 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f53e5ffb640 10 client.15942 handle_command_reply: tid=0 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.803+0000 7f5407408740 1 client.15942 shutdown 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 msgr2=0x56167c8ed110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 0x56167c8ed110 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f53f802c7b0 tx=0x7f53f802cbb0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 shutdown_connections 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x56167c95ab50 0x56167c95cf70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56167c8ee0d0 0x56167c8ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56167c8de240 0x56167c8ed110 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f53e5ffb640 0 client.15942 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 --2- 172.21.15.38:0/4147238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56167c8dd850 0x56167c8ecbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 >> 172.21.15.38:0/4147238593 conn(0x56167c8d8dc0 msgr2=0x56167c8cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 shutdown_connections 2024-04-22T01:14:13.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 1 -- 172.21.15.38:0/4147238593 wait complete. 2024-04-22T01:14:13.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.804+0000 7f5407408740 20 client.15942 trim_cache size 0 max 16384 2024-04-22T01:14:13.809 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53e80a2d70 msgr2=0x7f53e80a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53e80a2d70 0x7f53e80a5230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009270 tx=0x7f53f4097cc0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 msgr2=0x7f53fc1a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 0x7f53fc1a12c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f53f400eaa0 tx=0x7f53f400ef60 comp rx=0 tx=0).stop 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 shutdown_connections 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f53e80a2d70 0x7f53e80a5230 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53fc107460 0x7f53fc1a12c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53fc104b30 0x7f53fc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 --2- 172.21.15.38:0/480639257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53fc102200 0x7f53fc199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 >> 172.21.15.38:0/480639257 conn(0x7f53fc0fbbf0 msgr2=0x7f53fc101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 shutdown_connections 2024-04-22T01:14:13.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.805+0000 7f540473e640 1 -- 172.21.15.38:0/480639257 wait complete. 2024-04-22T01:14:13.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:13 smithi038 ceph-mon[28043]: pgmap v411: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:14:13.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T01:14:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:13 smithi111 ceph-mon[31057]: pgmap v411: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 -- 172.21.15.38:0/3178677682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c1022a0 msgr2=0x7f704c104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 --2- 172.21.15.38:0/3178677682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c1022a0 0x7f704c104690 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7048009900 tx=0x7f704802bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 -- 172.21.15.38:0/3178677682 shutdown_connections 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 --2- 172.21.15.38:0/3178677682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c107500 0x7f704c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 --2- 172.21.15.38:0/3178677682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f704c104bd0 0x7f704c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 --2- 172.21.15.38:0/3178677682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c1022a0 0x7f704c104690 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 -- 172.21.15.38:0/3178677682 >> 172.21.15.38:0/3178677682 conn(0x7f704c0fbbf0 msgr2=0x7f704c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 -- 172.21.15.38:0/3178677682 shutdown_connections 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.926+0000 7f70537d2640 1 -- 172.21.15.38:0/3178677682 wait complete. 2024-04-22T01:14:13.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 Processor -- start 2024-04-22T01:14:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 -- start start 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c104bd0 0x7f704c197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 0x7f704c198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f704c19d2b0 0x7f704c19f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f704c10d420 con 0x7f704c107500 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f704c10d2a0 con 0x7f704c104bd0 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f70537d2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f704c10d5a0 con 0x7f704c19d2b0 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f7050d6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 0x7f704c198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f7050d6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 0x7f704c198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48464/0 (socket says 172.21.15.38:48464) 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f7050d6d640 1 -- 172.21.15.38:0/1659004389 learned_addr learned my addr 172.21.15.38:0/1659004389 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.927+0000 7f7051d6f640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f704c19d2b0 0x7f704c19f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f705156e640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c104bd0 0x7f704c197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f7050d6d640 1 -- 172.21.15.38:0/1659004389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f704c19d2b0 msgr2=0x7f704c19f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f7050d6d640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f704c19d2b0 0x7f704c19f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f7050d6d640 1 -- 172.21.15.38:0/1659004389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c104bd0 msgr2=0x7f704c197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f7050d6d640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c104bd0 0x7f704c197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f7050d6d640 1 -- 172.21.15.38:0/1659004389 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c19fd40 con 0x7f704c107500 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f7050d6d640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 0x7f704c198290 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f703c00e9f0 tx=0x7f703c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f70427fc640 1 -- 172.21.15.38:0/1659004389 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c00ccc0 con 0x7f704c107500 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f70427fc640 1 -- 172.21.15.38:0/1659004389 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f703c005500 con 0x7f704c107500 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704c19ffd0 con 0x7f704c107500 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f70427fc640 1 -- 172.21.15.38:0/1659004389 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c010690 con 0x7f704c107500 2024-04-22T01:14:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.928+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f704c1a0440 con 0x7f704c107500 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 Processor -- start 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 -- start start 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e480870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 0x562a8e485940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e485fb0 0x562a8e488440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562a8e3e1530 con 0x562a8e480e40 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562a8e389830 con 0x562a8e485fb0 2024-04-22T01:14:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70563b6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562a8e3dd4f0 con 0x562a8e480450 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f70427fc640 1 -- 172.21.15.38:0/1659004389 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f703c0046c0 con 0x7f704c107500 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f705156e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 0x562a8e485940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f705156e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 0x562a8e485940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48466/0 (socket says 172.21.15.38:48466) 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f705156e640 1 -- 172.21.15.38:0/2304426912 learned_addr learned my addr 172.21.15.38:0/2304426912 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f7051d6f640 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e480870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.932+0000 7f7050d6d640 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e485fb0 0x562a8e488440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f705156e640 1 -- 172.21.15.38:0/2304426912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 msgr2=0x562a8e480870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f705156e640 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e480870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f705156e640 1 -- 172.21.15.38:0/2304426912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e485fb0 msgr2=0x562a8e488440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f705156e640 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e485fb0 0x562a8e488440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f705156e640 1 -- 172.21.15.38:0/2304426912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a8e3ad2c0 con 0x562a8e480e40 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f705156e640 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 0x562a8e485940 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7048000c00 tx=0x7f7048009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5bb1163a1fba873 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f70377fe640 1 -- 172.21.15.38:0/2304426912 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048053070 con 0x562a8e480e40 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f70377fe640 1 -- 172.21.15.38:0/2304426912 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f704802cb80 con 0x562a8e480e40 2024-04-22T01:14:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.933+0000 7f70377fe640 1 -- 172.21.15.38:0/2304426912 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70480046c0 con 0x562a8e480e40 2024-04-22T01:14:13.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70427fc640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f70380a2ca0 0x7f70380a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 -- 172.21.15.38:0/2304426912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 msgr2=0x562a8e485940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 0x562a8e485940 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7048000c00 tx=0x7f7048009560 comp rx=0 tx=0).stop 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f7051d6f640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f70380a2ca0 0x7f70380a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 -- 172.21.15.38:0/2304426912 shutdown_connections 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e485fb0 0x562a8e488440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480e40 0x562a8e485940 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 --2- 172.21.15.38:0/2304426912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e480870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 -- 172.21.15.38:0/2304426912 >> 172.21.15.38:0/2304426912 conn(0x562a8e4775b0 msgr2=0x562a8e4779b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 -- 172.21.15.38:0/2304426912 shutdown_connections 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 -- 172.21.15.38:0/2304426912 wait complete. 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.934+0000 7f70563b6740 1 -- start start 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70427fc640 1 -- 172.21.15.38:0/1659004389 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f703c0cfa80 con 0x7f704c107500 2024-04-22T01:14:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f7051d6f640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f70380a2ca0 0x7f70380a5160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x562a8e4816a0 tx=0x7f7044009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 Processor -- start 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480450 0x562a8e48a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480e40 0x562a8e48ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e491b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562a8e488f00 con 0x562a8e485fb0 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562a8e39cf10 con 0x562a8e480450 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562a8e489080 con 0x562a8e480e40 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f70563b6740 10 client.0 _ll_register_callbacks cb 0x562a8e0f4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f7051d6f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e491b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f7050d6d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480e40 0x562a8e48ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f7051d6f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e491b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48478/0 (socket says 172.21.15.38:48478) 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f7051d6f640 1 -- 172.21.15.38:0/4284499020 learned_addr learned my addr 172.21.15.38:0/4284499020 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.935+0000 7f705156e640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480450 0x562a8e48a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7050d6d640 1 -- 172.21.15.38:0/4284499020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480450 msgr2=0x562a8e48a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7050d6d640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480450 0x562a8e48a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7050d6d640 1 -- 172.21.15.38:0/4284499020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 msgr2=0x562a8e491b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7050d6d640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e491b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7050d6d640 1 -- 172.21.15.38:0/4284499020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a8e36ec80 con 0x562a8e480e40 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f705156e640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480450 0x562a8e48a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7051d6f640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e491b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7050d6d640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480e40 0x562a8e48ab10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f703c0cd020 tx=0x7f703c0d3080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f8de3527a58b86e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7034ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7034ff9640 1 -- 172.21.15.38:0/4284499020 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c09b8b0 con 0x562a8e480e40 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f704802cb80 con 0x562a8e480e40 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.936+0000 7f7034ff9640 1 -- 172.21.15.38:0/4284499020 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f703c0a4750 con 0x562a8e480e40 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f7034ff9640 1 -- 172.21.15.38:0/4284499020 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f703c0056b0 con 0x562a8e480e40 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 10 client.25366 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a8e0f53f0 con 0x562a8e480e40 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f7034ff9640 1 -- 172.21.15.38:0/4284499020 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f704802cb80 con 0x562a8e480e40 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f7034ff9640 1 -- 172.21.15.38:0/4284499020 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f703c09b350 con 0x562a8e480e40 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 10 client.25366 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 10 client.25366 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 20 client.25366 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4964a0 0x562a8e4988d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 4 client.25366 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a8e3eea70 con 0x562a8e4964a0 2024-04-22T01:14:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.937+0000 7f705156e640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4964a0 0x562a8e4988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.938+0000 7f705156e640 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4964a0 0x562a8e4988d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2979e907df7d2228 server_cookie=5830cb3cf61c57b1 in_seq=0 out_seq=0 2024-04-22T01:14:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.938+0000 7f7034ff9640 10 client.25366 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f7034ff9640 1 -- 172.21.15.38:0/4284499020 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a8e3eea70 con 0x562a8e4964a0 2024-04-22T01:14:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f7034ff9640 10 client.25366 handle_command_reply: tid=0 2024-04-22T01:14:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 client.25366 shutdown 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480e40 msgr2=0x562a8e48ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480e40 0x562a8e48ab10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f703c0cd020 tx=0x7f703c0d3080 comp rx=0 tx=0).stop 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 shutdown_connections 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4964a0 0x562a8e4988d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e491b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480e40 0x562a8e48ab10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f7034ff9640 0 client.25366 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 --2- 172.21.15.38:0/4284499020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480450 0x562a8e48a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 >> 172.21.15.38:0/4284499020 conn(0x562a8e4775b0 msgr2=0x562a8e476850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 shutdown_connections 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 1 -- 172.21.15.38:0/4284499020 wait complete. 2024-04-22T01:14:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.939+0000 7f70563b6740 20 client.25366 trim_cache size 0 max 16384 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 Processor -- start 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 -- start start 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 0x562a8e4766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e491b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e485fb0 0x562a8e49d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f704c073330 con 0x562a8e480450 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f704c1a1be0 con 0x562a8e480e40 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f70563b6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f704c1a0770 con 0x562a8e485fb0 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.966+0000 7f705156e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 0x562a8e4766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 0x562a8e4766d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48492/0 (socket says 172.21.15.38:48492) 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 -- 172.21.15.38:0/3020567907 learned_addr learned my addr 172.21.15.38:0/3020567907 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f7050d6d640 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e491b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f7051d6f640 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e485fb0 0x562a8e49d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 -- 172.21.15.38:0/3020567907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e485fb0 msgr2=0x562a8e49d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e485fb0 0x562a8e49d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 -- 172.21.15.38:0/3020567907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 msgr2=0x562a8e491b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e491b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 -- 172.21.15.38:0/3020567907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a8e1bf7d0 con 0x562a8e480450 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f7050d6d640 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e491b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f705156e640 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 0x562a8e4766d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7048005940 tx=0x7f704802c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fa9c9bfa95d0af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f7036ffd640 1 -- 172.21.15.38:0/3020567907 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048053040 con 0x562a8e480450 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f7036ffd640 1 -- 172.21.15.38:0/3020567907 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7048005020 con 0x562a8e480450 2024-04-22T01:14:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.967+0000 7f7036ffd640 1 -- 172.21.15.38:0/3020567907 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048004920 con 0x562a8e480450 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 -- 172.21.15.38:0/3020567907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 msgr2=0x562a8e4766d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 0x562a8e4766d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7048005940 tx=0x7f704802c880 comp rx=0 tx=0).stop 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 -- 172.21.15.38:0/3020567907 shutdown_connections 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e485fb0 0x562a8e49d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e491b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 --2- 172.21.15.38:0/3020567907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e480450 0x562a8e4766d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 -- 172.21.15.38:0/3020567907 >> 172.21.15.38:0/3020567907 conn(0x562a8e4775b0 msgr2=0x562a8e475fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.968+0000 7f70563b6740 1 -- 172.21.15.38:0/3020567907 shutdown_connections 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.969+0000 7f70563b6740 1 -- 172.21.15.38:0/3020567907 wait complete. 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.969+0000 7f70563b6740 1 -- start start 2024-04-22T01:14:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.969+0000 7f70563b6740 1 Processor -- start 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f70563b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e48f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f70563b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e48f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f70563b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e490670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f70563b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f704c1a1510 con 0x562a8e485fb0 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f705156e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e48f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f705156e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e48f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50880/0 (socket says 172.21.15.38:50880) 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f705156e640 1 -- 172.21.15.38:0/1054472467 learned_addr learned my addr 172.21.15.38:0/1054472467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f7051d6f640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e490670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f704c1a0e40 con 0x562a8e480e40 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.970+0000 7f7050d6d640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e48f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f704c0712f0 con 0x562a8e480450 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f70563b6740 10 client.0 _ll_register_callbacks cb 0x7f703c0d7040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f705156e640 1 -- 172.21.15.38:0/1054472467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 msgr2=0x562a8e48f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f705156e640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e48f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f705156e640 1 -- 172.21.15.38:0/1054472467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 msgr2=0x562a8e490670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f705156e640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e490670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f705156e640 1 -- 172.21.15.38:0/1054472467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a8e490c60 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f7051d6f640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e490670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f705156e640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e48f120 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7048000c00 tx=0x7f704802cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fff99c90b95f9d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f7034ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f7034ff9640 1 -- 172.21.15.38:0/1054472467 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048053040 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7048005020 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.971+0000 7f7034ff9640 1 -- 172.21.15.38:0/1054472467 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7048004260 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f7034ff9640 1 -- 172.21.15.38:0/1054472467 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f704806c050 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 10 client.25372 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a8e491ed0 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f7034ff9640 1 -- 172.21.15.38:0/1054472467 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7048005020 con 0x562a8e480450 2024-04-22T01:14:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f7034ff9640 1 -- 172.21.15.38:0/1054472467 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f70480046c0 con 0x562a8e480450 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 10 client.25372 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 10 client.25372 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 20 client.25372 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4fcd10 0x562a8e4ff130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 4 client.25372 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.972+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a8e3e8dc0 con 0x562a8e4fcd10 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f7050d6d640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4fcd10 0x562a8e4ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f7034ff9640 10 client.25372 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f7050d6d640 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4fcd10 0x562a8e4ff130 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b62742a1be74f0dd server_cookie=bc1c6c0f6b85f920 in_seq=0 out_seq=0 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f7034ff9640 1 -- 172.21.15.38:0/1054472467 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a8e3e8dc0 con 0x562a8e4fcd10 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f7034ff9640 10 client.25372 handle_command_reply: tid=0 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 client.25372 shutdown 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 msgr2=0x562a8e48f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e48f120 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7048000c00 tx=0x7f704802cc70 comp rx=0 tx=0).stop 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 shutdown_connections 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x562a8e4fcd10 0x562a8e4ff130 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562a8e485fb0 0x562a8e490670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f7034ff9640 0 client.25372 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562a8e480e40 0x562a8e48f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 --2- 172.21.15.38:0/1054472467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562a8e480450 0x562a8e48f120 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.973+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 >> 172.21.15.38:0/1054472467 conn(0x562a8e4775b0 msgr2=0x562a8e476850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 shutdown_connections 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70563b6740 1 -- 172.21.15.38:0/1054472467 wait complete. 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70563b6740 20 client.25372 trim_cache size 0 max 16384 2024-04-22T01:14:13.977 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f70380a2ca0 msgr2=0x7f70380a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70537d2640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f70380a2ca0 0x7f70380a5160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x562a8e4816a0 tx=0x7f7044009500 comp rx=0 tx=0).stop 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 msgr2=0x7f704c198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.974+0000 7f70537d2640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 0x7f704c198290 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f703c00e9f0 tx=0x7f703c00edf0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 shutdown_connections 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f70380a2ca0 0x7f70380a5160 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f704c19d2b0 0x7f704c19f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f704c107500 0x7f704c198290 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 --2- 172.21.15.38:0/1659004389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c104bd0 0x7f704c197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 >> 172.21.15.38:0/1659004389 conn(0x7f704c0fbbf0 msgr2=0x7f704c104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 shutdown_connections 2024-04-22T01:14:13.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:13.975+0000 7f70537d2640 1 -- 172.21.15.38:0/1659004389 wait complete. 2024-04-22T01:14:13.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/4272270307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df8100050 msgr2=0x7f7df8100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/4272270307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df8100050 0x7f7df8100470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009900 tx=0x7f7de802bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/4272270307 shutdown_connections 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/4272270307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8101350 0x7f7df81017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/4272270307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df81009b0 0x7f7df8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/4272270307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df8100050 0x7f7df8100470 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/4272270307 >> 172.21.15.38:0/4272270307 conn(0x7f7df80fbbf0 msgr2=0x7f7df80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/4272270307 shutdown_connections 2024-04-22T01:14:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.093+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/4272270307 wait complete. 2024-04-22T01:14:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 Processor -- start 2024-04-22T01:14:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 -- start start 2024-04-22T01:14:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8100050 0x7f7df8199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 0x7f7df819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df8101350 0x7f7df81a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810f5c0 con 0x7f7df81009b0 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810f440 con 0x7f7df8100050 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.094+0000 7f7dfe4fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810f740 con 0x7f7df8101350 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 0x7f7df819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 0x7f7df819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48522/0 (socket says 172.21.15.38:48522) 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 -- 172.21.15.38:0/3249894331 learned_addr learned my addr 172.21.15.38:0/3249894331 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8100050 0x7f7df8199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7dfca98640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df8101350 0x7f7df81a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 -- 172.21.15.38:0/3249894331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df8101350 msgr2=0x7f7df81a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df8101350 0x7f7df81a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 -- 172.21.15.38:0/3249894331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8100050 msgr2=0x7f7df8199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8100050 0x7f7df8199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 -- 172.21.15.38:0/3249894331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df81a1900 con 0x7f7df81009b0 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7dfca98640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df8101350 0x7f7df81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8100050 0x7f7df8199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 0x7f7df819a200 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00d950 tx=0x7f7dec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df57fa640 1 -- 172.21.15.38:0/3249894331 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec017070 con 0x7f7df81009b0 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df81a1bf0 con 0x7f7df81009b0 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df57fa640 1 -- 172.21.15.38:0/3249894331 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec009c30 con 0x7f7df81009b0 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.095+0000 7f7df57fa640 1 -- 172.21.15.38:0/3249894331 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec00f6b0 con 0x7f7df81009b0 2024-04-22T01:14:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.096+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81a2140 con 0x7f7df81009b0 2024-04-22T01:14:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.098+0000 7f7df57fa640 1 -- 172.21.15.38:0/3249894331 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f7dec0109d0 con 0x7f7df81009b0 2024-04-22T01:14:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.098+0000 7f7df57fa640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7de00a2c20 0x7f7de00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7df57fa640 1 -- 172.21.15.38:0/3249894331 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7dec0cf260 con 0x7f7df81009b0 2024-04-22T01:14:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 Processor -- start 2024-04-22T01:14:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7de00a2c20 0x7f7de00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 -- start start 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f2f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2fa30 0x563045f34530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 0x563045f37030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563045e94530 con 0x563045f34ba0 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563045e3c830 con 0x563045f2fa30 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7e011c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563045e904f0 con 0x563045f2f040 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7dfca98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 0x563045f37030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7dfca98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 0x563045f37030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48530/0 (socket says 172.21.15.38:48530) 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7dfca98640 1 -- 172.21.15.38:0/2480704384 learned_addr learned my addr 172.21.15.38:0/2480704384 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.099+0000 7f7df77fe640 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2fa30 0x563045f34530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7de00a2c20 0x7f7de00a50e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7de8036040 tx=0x7f7de8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dfca98640 1 -- 172.21.15.38:0/2480704384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 msgr2=0x563045f2f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dfca98640 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f2f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dfca98640 1 -- 172.21.15.38:0/2480704384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2fa30 msgr2=0x563045f34530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dfca98640 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2fa30 0x563045f34530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dfca98640 1 -- 172.21.15.38:0/2480704384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563045e602c0 con 0x563045f34ba0 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7df77fe640 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2fa30 0x563045f34530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dfca98640 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 0x563045f37030 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009f90 tx=0x7f7df000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be874e00501cd9f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dde7fc640 1 -- 172.21.15.38:0/2480704384 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0003b70 con 0x563045f34ba0 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dde7fc640 1 -- 172.21.15.38:0/2480704384 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df0003d10 con 0x563045f34ba0 2024-04-22T01:14:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.100+0000 7f7dde7fc640 1 -- 172.21.15.38:0/2480704384 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0011a30 con 0x563045f34ba0 2024-04-22T01:14:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 -- 172.21.15.38:0/2480704384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 msgr2=0x563045f37030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 0x563045f37030 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009f90 tx=0x7f7df000f7f0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 -- 172.21.15.38:0/2480704384 shutdown_connections 2024-04-22T01:14:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f34ba0 0x563045f37030 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2fa30 0x563045f34530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 --2- 172.21.15.38:0/2480704384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f2f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 -- 172.21.15.38:0/2480704384 >> 172.21.15.38:0/2480704384 conn(0x563045f2a5b0 msgr2=0x563045f2a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 -- 172.21.15.38:0/2480704384 shutdown_connections 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.101+0000 7f7e011c4740 1 -- 172.21.15.38:0/2480704384 wait complete. 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 -- start start 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 Processor -- start 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f39340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f39880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f34ba0 0x563045f40850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563045f37b50 con 0x563045f2fa30 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563045e4ff10 con 0x563045f34ba0 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7df7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f39340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7df77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f39880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7df7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f39340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50930/0 (socket says 172.21.15.38:50930) 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7df7fff640 1 -- 172.21.15.38:0/4083730696 learned_addr learned my addr 172.21.15.38:0/4083730696 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.102+0000 7f7e011c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563045f37cd0 con 0x563045f2f040 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7e011c4740 10 client.0 _ll_register_callbacks cb 0x563045ba7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7dfca98640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f34ba0 0x563045f40850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df7fff640 1 -- 172.21.15.38:0/4083730696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f34ba0 msgr2=0x563045f40850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df7fff640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f34ba0 0x563045f40850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df7fff640 1 -- 172.21.15.38:0/4083730696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 msgr2=0x563045f39880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df7fff640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f39880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df7fff640 1 -- 172.21.15.38:0/4083730696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563045e21c80 con 0x563045f2f040 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7dfca98640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f34ba0 0x563045f40850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df77fe640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f39880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7df7fff640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f39340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7de80096c0 tx=0x7f7de80042c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fdf90761d8cc81a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7dd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7dd77fe640 1 -- 172.21.15.38:0/4083730696 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de80044b0 con 0x563045f2f040 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7dd77fe640 1 -- 172.21.15.38:0/4083730696 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7de804a460 con 0x563045f2f040 2024-04-22T01:14:14.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.103+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df0003d10 con 0x563045f2f040 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7dd77fe640 1 -- 172.21.15.38:0/4083730696 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7de8058050 con 0x563045f2f040 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 10 client.25378 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563045ba83f0 con 0x563045f2f040 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7dd77fe640 1 -- 172.21.15.38:0/4083730696 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df0003d10 con 0x563045f2f040 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7dd77fe640 1 -- 172.21.15.38:0/4083730696 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7de803f710 con 0x563045f2f040 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 10 client.25378 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 10 client.25378 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 20 client.25378 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f45110 0x563045f475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 4 client.25378 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:14.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563045ea1a70 con 0x563045f45110 2024-04-22T01:14:14.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.104+0000 7f7df77fe640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f45110 0x563045f475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.105+0000 7f7dd77fe640 10 client.25378 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:14.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.105+0000 7f7df77fe640 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f45110 0x563045f475d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a372b75cf31553f server_cookie=d755ad40966f8383 in_seq=0 out_seq=0 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7dd77fe640 1 -- 172.21.15.38:0/4083730696 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563045ea1a70 con 0x563045f45110 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7dd77fe640 10 client.25378 handle_command_reply: tid=0 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 client.25378 shutdown 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 msgr2=0x563045f39340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f39340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7de80096c0 tx=0x7f7de80042c0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 shutdown_connections 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f45110 0x563045f475d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f34ba0 0x563045f40850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f39880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7dd77fe640 0 client.25378 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 --2- 172.21.15.38:0/4083730696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2f040 0x563045f39340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 >> 172.21.15.38:0/4083730696 conn(0x563045f2a5b0 msgr2=0x563045f29850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 shutdown_connections 2024-04-22T01:14:14.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 1 -- 172.21.15.38:0/4083730696 wait complete. 2024-04-22T01:14:14.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.106+0000 7f7e011c4740 20 client.25378 trim_cache size 0 max 16384 2024-04-22T01:14:14.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 Processor -- start 2024-04-22T01:14:14.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 -- start start 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2a5b0 0x563045f30290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f40810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f307d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81ac890 con 0x563045f2fa30 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81ae3d0 con 0x563045f2f040 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7e011c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81acf60 con 0x563045f2a5b0 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7dfca98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7dfca98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f307d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48552/0 (socket says 172.21.15.38:48552) 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7dfca98640 1 -- 172.21.15.38:0/1825335776 learned_addr learned my addr 172.21.15.38:0/1825335776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7df77fe640 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f40810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.120+0000 7f7df7fff640 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2a5b0 0x563045f30290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dfca98640 1 -- 172.21.15.38:0/1825335776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2a5b0 msgr2=0x563045f30290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dfca98640 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2a5b0 0x563045f30290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dfca98640 1 -- 172.21.15.38:0/1825335776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 msgr2=0x563045f40810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dfca98640 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f40810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dfca98640 1 -- 172.21.15.38:0/1825335776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563045c727d0 con 0x563045f2fa30 2024-04-22T01:14:14.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dfca98640 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f307d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x563045f2b520 tx=0x7f7df0005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8f8a18012b9334 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dddffb640 1 -- 172.21.15.38:0/1825335776 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0004a00 con 0x563045f2fa30 2024-04-22T01:14:14.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dddffb640 1 -- 172.21.15.38:0/1825335776 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df0003e50 con 0x563045f2fa30 2024-04-22T01:14:14.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.121+0000 7f7dddffb640 1 -- 172.21.15.38:0/1825335776 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00047a0 con 0x563045f2fa30 2024-04-22T01:14:14.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 -- 172.21.15.38:0/1825335776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 msgr2=0x563045f307d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f307d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x563045f2b520 tx=0x7f7df0005df0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 -- 172.21.15.38:0/1825335776 shutdown_connections 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2fa30 0x563045f307d0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f40810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 --2- 172.21.15.38:0/1825335776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2a5b0 0x563045f30290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 -- 172.21.15.38:0/1825335776 >> 172.21.15.38:0/1825335776 conn(0x563045f346e0 msgr2=0x563045f2e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 -- 172.21.15.38:0/1825335776 shutdown_connections 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 -- 172.21.15.38:0/1825335776 wait complete. 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.122+0000 7f7e011c4740 1 -- start start 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 Processor -- start 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 0x563045f3e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f3e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2fa30 0x563045f3f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81add00 con 0x563045f2a5b0 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81ad630 con 0x563045f2f040 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81aa850 con 0x563045f2fa30 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 0x563045f3e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7e011c4740 10 client.0 _ll_register_callbacks cb 0x7f7df00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 0x563045f3e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48558/0 (socket says 172.21.15.38:48558) 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 -- 172.21.15.38:0/3350346128 learned_addr learned my addr 172.21.15.38:0/3350346128 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f3e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7dfca98640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2fa30 0x563045f3f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 -- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2fa30 msgr2=0x563045f3f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2fa30 0x563045f3f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 -- 172.21.15.38:0/3350346128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 msgr2=0x563045f3e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f3e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df7fff640 1 -- 172.21.15.38:0/3350346128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563045f3fcf0 con 0x563045f2a5b0 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f3e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.123+0000 7f7dfca98640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2fa30 0x563045f3f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:14.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7df7fff640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 0x563045f3e1b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7de802f020 tx=0x7f7de8003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7572cf83b36248e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7dd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7dd77fe640 1 -- 172.21.15.38:0/3350346128 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de8034800 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7dd77fe640 1 -- 172.21.15.38:0/3350346128 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f7de80349a0 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df0003e50 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7dd77fe640 1 -- 172.21.15.38:0/3350346128 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7de806d050 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7e011c4740 10 client.15978 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563045fab990 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.124+0000 7f7dd77fe640 1 -- 172.21.15.38:0/3350346128 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de804f070 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7dd77fe640 1 -- 172.21.15.38:0/3350346128 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7de804a940 con 0x563045f2a5b0 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7e011c4740 10 client.15978 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7e011c4740 10 client.15978 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7e011c4740 20 client.15978 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7e011c4740 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f34ba0 0x563045fae390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7e011c4740 4 client.15978 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7df0003e50 con 0x563045f34ba0 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f34ba0 0x563045fae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7dd77fe640 10 client.15978 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.125+0000 7f7df77fe640 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f34ba0 0x563045fae390 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aeaf0c0611c34c1b server_cookie=e76e531dda6fde8d in_seq=0 out_seq=0 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7dd77fe640 1 -- 172.21.15.38:0/3350346128 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7df0003e50 con 0x563045f34ba0 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7dd77fe640 10 client.15978 handle_command_reply: tid=0 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 client.15978 shutdown 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 msgr2=0x563045f3e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 0x563045f3e1b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7de802f020 tx=0x7f7de8003fd0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 shutdown_connections 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563045f34ba0 0x563045fae390 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7dd77fe640 0 client.15978 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563045f2fa30 0x563045f3f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563045f2f040 0x563045f3e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 --2- 172.21.15.38:0/3350346128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563045f2a5b0 0x563045f3e1b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 >> 172.21.15.38:0/3350346128 conn(0x563045f346e0 msgr2=0x563045f28fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 shutdown_connections 2024-04-22T01:14:14.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 1 -- 172.21.15.38:0/3350346128 wait complete. 2024-04-22T01:14:14.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.126+0000 7f7e011c4740 20 client.15978 trim_cache size 0 max 16384 2024-04-22T01:14:14.130 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7de00a2c20 msgr2=0x7f7de00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7de00a2c20 0x7f7de00a50e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7de8036040 tx=0x7f7de8000bc0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 msgr2=0x7f7df819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 0x7f7df819a200 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00d950 tx=0x7f7dec00de10 comp rx=0 tx=0).stop 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 shutdown_connections 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f7de00a2c20 0x7f7de00a50e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7df8101350 0x7f7df81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7df81009b0 0x7f7df819a200 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 --2- 172.21.15.38:0/3249894331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7df8100050 0x7f7df8199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 >> 172.21.15.38:0/3249894331 conn(0x7f7df80fbbf0 msgr2=0x7f7df80fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 shutdown_connections 2024-04-22T01:14:14.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.127+0000 7f7dfe4fb640 1 -- 172.21.15.38:0/3249894331 wait complete. 2024-04-22T01:14:14.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 -- 172.21.15.38:0/58955841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68102980 msgr2=0x7f8f68102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 --2- 172.21.15.38:0/58955841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68102980 0x7f8f68102e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009860 tx=0x7f8f5c02bb70 comp rx=0 tx=0).stop 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 -- 172.21.15.38:0/58955841 shutdown_connections 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 --2- 172.21.15.38:0/58955841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68103340 0x7f8f68109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 --2- 172.21.15.38:0/58955841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68102980 0x7f8f68102e00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 --2- 172.21.15.38:0/58955841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 -- 172.21.15.38:0/58955841 >> 172.21.15.38:0/58955841 conn(0x7f8f680fcd20 msgr2=0x7f8f680ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 -- 172.21.15.38:0/58955841 shutdown_connections 2024-04-22T01:14:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.246+0000 7f8f6e385640 1 -- 172.21.15.38:0/58955841 wait complete. 2024-04-22T01:14:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 Processor -- start 2024-04-22T01:14:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 -- start start 2024-04-22T01:14:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68102980 0x7f8f68197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68103340 0x7f8f6819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810d410 con 0x7f8f68101790 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810d290 con 0x7f8f68103340 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f6e385640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810d590 con 0x7f8f68102980 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.247+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48576/0 (socket says 172.21.15.38:48576) 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68102980 0x7f8f68197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f67fff640 1 -- 172.21.15.38:0/883077415 learned_addr learned my addr 172.21.15.38:0/883077415 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f6c922640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68103340 0x7f8f6819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 -- 172.21.15.38:0/883077415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68103340 msgr2=0x7f8f6819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68103340 0x7f8f6819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 -- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 msgr2=0x7f8f68197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 -- 172.21.15.38:0/883077415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6819f6d0 con 0x7f8f68102980 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f67fff640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f677fe640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68102980 0x7f8f68197ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c02c0b0 tx=0x7f8f5c032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.248+0000 7f8f657fa640 1 -- 172.21.15.38:0/883077415 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0044d0 con 0x7f8f68102980 2024-04-22T01:14:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.249+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f68073970 con 0x7f8f68102980 2024-04-22T01:14:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.249+0000 7f8f657fa640 1 -- 172.21.15.38:0/883077415 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f5c02c9f0 con 0x7f8f68102980 2024-04-22T01:14:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.249+0000 7f8f657fa640 1 -- 172.21.15.38:0/883077415 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c003850 con 0x7f8f68102980 2024-04-22T01:14:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.249+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f68073eb0 con 0x7f8f68102980 2024-04-22T01:14:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f657fa640 1 -- 172.21.15.38:0/883077415 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f8f5c032940 con 0x7f8f68102980 2024-04-22T01:14:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 Processor -- start 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 -- start start 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fdfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f116f3c530 con 0x55f116fe0240 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f116ee4830 con 0x55f116fdf850 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.252+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f116f384f0 con 0x55f116fe53b0 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48588/0 (socket says 172.21.15.38:48588) 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 -- 172.21.15.38:0/2194233243 learned_addr learned my addr 172.21.15.38:0/2194233243 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f677fe640 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f6c922640 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fdfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f657fa640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8f500a2e40 0x7f8f500a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 -- 172.21.15.38:0/2194233243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 msgr2=0x55f116fe7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f657fa640 1 -- 172.21.15.38:0/883077415 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f8f5c0f4f00 con 0x7f8f68102980 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 -- 172.21.15.38:0/2194233243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 msgr2=0x55f116fdfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fdfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 -- 172.21.15.38:0/2194233243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f116f082c0 con 0x55f116fe0240 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f6c922640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8f500a2e40 0x7f8f500a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f677fe640 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.253+0000 7f8f67fff640 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe4d40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009fd0 tx=0x7f8f58011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86afe6c771f321de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f4e7fc640 1 -- 172.21.15.38:0/2194233243 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5801e070 con 0x55f116fe0240 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f6c922640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8f500a2e40 0x7f8f500a5300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55f116fe0aa0 tx=0x7f8f60004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f4e7fc640 1 -- 172.21.15.38:0/2194233243 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f5800fb20 con 0x55f116fe0240 2024-04-22T01:14:14.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f4e7fc640 1 -- 172.21.15.38:0/2194233243 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58019ce0 con 0x55f116fe0240 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2194233243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 msgr2=0x55f116fe4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe4d40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009fd0 tx=0x7f8f58011a50 comp rx=0 tx=0).stop 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2194233243 shutdown_connections 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe4d40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2194233243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fdfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2194233243 >> 172.21.15.38:0/2194233243 conn(0x55f116fd25b0 msgr2=0x55f116fd29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2194233243 shutdown_connections 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.254+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2194233243 wait complete. 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.255+0000 7f8f70f6a740 1 -- start start 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.255+0000 7f8f70f6a740 1 Processor -- start 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.255+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fe9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.255+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe53b0 0x55f116ff0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48590/0 (socket says 172.21.15.38:48590) 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 -- 172.21.15.38:0/459535319 learned_addr learned my addr 172.21.15.38:0/459535319 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f116fe8360 con 0x55f116fe0240 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f67fff640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fe9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f6c922640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe53b0 0x55f116ff0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f116ef7f10 con 0x55f116fe53b0 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f116fe84e0 con 0x55f116fdf850 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f70f6a740 10 client.0 _ll_register_callbacks cb 0x55f116c4fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 -- 172.21.15.38:0/459535319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 msgr2=0x55f116fe9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fe9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 -- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe53b0 msgr2=0x55f116ff0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe53b0 0x55f116ff0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f677fe640 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f116ec9c80 con 0x55f116fe0240 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.256+0000 7f8f6c922640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe53b0 0x55f116ff0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f677fe640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe9fa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000c00 tx=0x7f8f5c02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11de6b717b46774a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f477fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f477fe640 1 -- 172.21.15.38:0/459535319 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0c0ca0 con 0x55f116fe0240 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f5800fb20 con 0x55f116fe0240 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f477fe640 1 -- 172.21.15.38:0/459535319 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f8f5c0c15a0 con 0x55f116fe0240 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f477fe640 1 -- 172.21.15.38:0/459535319 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8f5c0c1a40 con 0x55f116fe0240 2024-04-22T01:14:14.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f70f6a740 10 client.15990 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.257+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f116c503f0 con 0x55f116fe0240 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f477fe640 1 -- 172.21.15.38:0/459535319 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f5800fb20 con 0x55f116fe0240 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f477fe640 1 -- 172.21.15.38:0/459535319 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8f5c043070 con 0x55f116fe0240 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f70f6a740 10 client.15990 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f70f6a740 10 client.15990 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f70f6a740 20 client.15990 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f116ff7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f70f6a740 4 client.15990 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f116f49a70 con 0x55f116ff57d0 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.258+0000 7f8f67fff640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f116ff7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.259+0000 7f8f67fff640 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f116ff7c50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e34dd8788a5646c server_cookie=4e387202fea9d4b3 in_seq=0 out_seq=0 2024-04-22T01:14:14.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.259+0000 7f8f477fe640 10 client.15990 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.259+0000 7f8f477fe640 1 -- 172.21.15.38:0/459535319 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f116f49a70 con 0x55f116ff57d0 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.259+0000 7f8f477fe640 10 client.15990 handle_command_reply: tid=0 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.259+0000 7f8f70f6a740 1 client.15990 shutdown 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 msgr2=0x55f116fe9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe9fa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000c00 tx=0x7f8f5c02bb70 comp rx=0 tx=0).stop 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 shutdown_connections 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f116ff7c50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe53b0 0x55f116ff0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f477fe640 0 client.15990 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116fe9fa0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/459535319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fe9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 >> 172.21.15.38:0/459535319 conn(0x55f116fd25b0 msgr2=0x55f116fd1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 shutdown_connections 2024-04-22T01:14:14.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 1 -- 172.21.15.38:0/459535319 wait complete. 2024-04-22T01:14:14.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.260+0000 7f8f70f6a740 20 client.15990 trim_cache size 0 max 16384 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.286+0000 7f8f70f6a740 1 Processor -- start 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.286+0000 7f8f70f6a740 1 -- start start 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fe9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116ff0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681a8320 con 0x55f116fe0240 2024-04-22T01:14:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116ff0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681a9e60 con 0x55f116fdf850 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681a89f0 con 0x55f116fe53b0 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116ff0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48598/0 (socket says 172.21.15.38:48598) 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 -- 172.21.15.38:0/465873283 learned_addr learned my addr 172.21.15.38:0/465873283 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f67fff640 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fe9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f6c922640 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 -- 172.21.15.38:0/465873283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 msgr2=0x55f116fe0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 -- 172.21.15.38:0/465873283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 msgr2=0x55f116fe9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fe9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f677fe640 1 -- 172.21.15.38:0/465873283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f116d1a7d0 con 0x55f116fe0240 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.287+0000 7f8f6c922640 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.288+0000 7f8f677fe640 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116ff0f30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0bdd10 tx=0x7f8f5c038170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4fc6bd8bc475b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.288+0000 7f8f4dffb640 1 -- 172.21.15.38:0/465873283 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0c0ca0 con 0x55f116fe0240 2024-04-22T01:14:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.288+0000 7f8f4dffb640 1 -- 172.21.15.38:0/465873283 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f5c0c9b20 con 0x55f116fe0240 2024-04-22T01:14:14.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.288+0000 7f8f4dffb640 1 -- 172.21.15.38:0/465873283 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0c15a0 con 0x55f116fe0240 2024-04-22T01:14:14.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 -- 172.21.15.38:0/465873283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 msgr2=0x55f116ff0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116ff0f30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0bdd10 tx=0x7f8f5c038170 comp rx=0 tx=0).stop 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 -- 172.21.15.38:0/465873283 shutdown_connections 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fe53b0 0x55f116fe0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe0240 0x55f116ff0f30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/465873283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fdf850 0x55f116fe9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 -- 172.21.15.38:0/465873283 >> 172.21.15.38:0/465873283 conn(0x55f116fd25b0 msgr2=0x55f116fd60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 -- 172.21.15.38:0/465873283 shutdown_connections 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 -- 172.21.15.38:0/465873283 wait complete. 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 -- start start 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.289+0000 7f8f70f6a740 1 Processor -- start 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 0x55f116feeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 0x55f116fefab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681a9790 con 0x55f116fe53b0 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681a90c0 con 0x55f116fe0240 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681a62e0 con 0x55f116fdf850 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f70f6a740 10 client.0 _ll_register_callbacks cb 0x7f8f5c0c1200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 0x55f116fefab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 0x55f116feeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 0x55f116fefab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48602/0 (socket says 172.21.15.38:48602) 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 -- 172.21.15.38:0/2894094083 learned_addr learned my addr 172.21.15.38:0/2894094083 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 0x55f116feeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60584/0 (socket says 172.21.15.38:60584) 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f67fff640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 -- 172.21.15.38:0/2894094083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 msgr2=0x55f116fee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 -- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 msgr2=0x55f116feeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 0x55f116feeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f6c922640 1 -- 172.21.15.38:0/2894094083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f116ff00a0 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.290+0000 7f8f677fe640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 0x55f116feeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f6c922640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 0x55f116fefab0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000ff10 tx=0x7f8f60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42be85f3499f5c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f477fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f477fe640 1 -- 172.21.15.38:0/2894094083 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6002b420 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f477fe640 1 -- 172.21.15.38:0/2894094083 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f8f6002b5c0 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f5c0c9b20 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f477fe640 1 -- 172.21.15.38:0/2894094083 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8f6002b7c0 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f70f6a740 10 client.15996 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f116ff1380 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.291+0000 7f8f477fe640 1 -- 172.21.15.38:0/2894094083 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f5c0c9b20 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f477fe640 1 -- 172.21.15.38:0/2894094083 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8f6001c040 con 0x55f116fe53b0 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f70f6a740 10 client.15996 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f70f6a740 10 client.15996 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:14:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f70f6a740 20 client.15996 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f11705e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f70f6a740 4 client.15996 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f116f43dc0 con 0x55f116ff57d0 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f67fff640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f11705e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f67fff640 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f11705e660 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5ced600654febeb server_cookie=57219611df81f72d in_seq=0 out_seq=0 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.292+0000 7f8f477fe640 10 client.15996 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f477fe640 1 -- 172.21.15.38:0/2894094083 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f116f43dc0 con 0x55f116ff57d0 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f477fe640 10 client.15996 handle_command_reply: tid=0 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 client.15996 shutdown 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 msgr2=0x55f116fefab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 0x55f116fefab0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000ff10 tx=0x7f8f60004a40 comp rx=0 tx=0).stop 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 shutdown_connections 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55f116ff57d0 0x55f11705e660 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f477fe640 0 client.15996 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:14.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f116fe53b0 0x55f116fefab0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f116fe0240 0x55f116feeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 --2- 172.21.15.38:0/2894094083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f116fdf850 0x55f116fee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 >> 172.21.15.38:0/2894094083 conn(0x55f116fd25b0 msgr2=0x55f116fd60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 shutdown_connections 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 1 -- 172.21.15.38:0/2894094083 wait complete. 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.293+0000 7f8f70f6a740 20 client.15996 trim_cache size 0 max 16384 2024-04-22T01:14:14.297 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8f500a2e40 msgr2=0x7f8f500a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8f500a2e40 0x7f8f500a5300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55f116fe0aa0 tx=0x7f8f60004ab0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68102980 msgr2=0x7f8f68197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68102980 0x7f8f68197ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c02c0b0 tx=0x7f8f5c032430 comp rx=0 tx=0).stop 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 shutdown_connections 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8f500a2e40 0x7f8f500a5300 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f68103340 0x7f8f6819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68102980 0x7f8f68197ff0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 --2- 172.21.15.38:0/883077415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f68101790 0x7f8f68197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 >> 172.21.15.38:0/883077415 conn(0x7f8f680fcd20 msgr2=0x7f8f68105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 shutdown_connections 2024-04-22T01:14:14.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.294+0000 7f8f6e385640 1 -- 172.21.15.38:0/883077415 wait complete. 2024-04-22T01:14:14.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T01:14:14.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 -- 172.21.15.38:0/1561586843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 msgr2=0x7f6684101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 --2- 172.21.15.38:0/1561586843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f6684101cf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6674009900 tx=0x7f667402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:14.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 -- 172.21.15.38:0/1561586843 shutdown_connections 2024-04-22T01:14:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 --2- 172.21.15.38:0/1561586843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6684102230 0x7f6684108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 --2- 172.21.15.38:0/1561586843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f6684101cf0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 --2- 172.21.15.38:0/1561586843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6684100680 0x7f6684100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 -- 172.21.15.38:0/1561586843 >> 172.21.15.38:0/1561586843 conn(0x7f66840fbbf0 msgr2=0x7f66840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 -- 172.21.15.38:0/1561586843 shutdown_connections 2024-04-22T01:14:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.414+0000 7f668b2fb640 1 -- 172.21.15.38:0/1561586843 wait complete. 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 Processor -- start 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 -- start start 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6684100680 0x7f6684069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f668419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6684102230 0x7f668419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f668410c300 con 0x7f6684101870 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f668410c180 con 0x7f6684102230 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f668b2fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f668410c480 con 0x7f6684100680 2024-04-22T01:14:14.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f6688896640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f668419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f6688896640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f668419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48612/0 (socket says 172.21.15.38:48612) 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f6688896640 1 -- 172.21.15.38:0/2377066479 learned_addr learned my addr 172.21.15.38:0/2377066479 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f6689097640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6684100680 0x7f6684069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.415+0000 7f6689898640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6684102230 0x7f668419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f6688896640 1 -- 172.21.15.38:0/2377066479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6684100680 msgr2=0x7f6684069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f6688896640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6684100680 0x7f6684069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f6688896640 1 -- 172.21.15.38:0/2377066479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6684102230 msgr2=0x7f668419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f6688896640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6684102230 0x7f668419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f6688896640 1 -- 172.21.15.38:0/2377066479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66841965d0 con 0x7f6684101870 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f6688896640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f668419c800 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6674009230 tx=0x7f667402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f667a7fc640 1 -- 172.21.15.38:0/2377066479 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674041070 con 0x7f6684101870 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6684196860 con 0x7f6684101870 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f667a7fc640 1 -- 172.21.15.38:0/2377066479 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6674004450 con 0x7f6684101870 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f667a7fc640 1 -- 172.21.15.38:0/2377066479 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674003e20 con 0x7f6684101870 2024-04-22T01:14:14.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.416+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6684196da0 con 0x7f6684101870 2024-04-22T01:14:14.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.419+0000 7f667a7fc640 1 -- 172.21.15.38:0/2377066479 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f6674044070 con 0x7f6684101870 2024-04-22T01:14:14.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 Processor -- start 2024-04-22T01:14:14.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 -- start start 2024-04-22T01:14:14.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f667a7fc640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f66700a2c20 0x7f66700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc84b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 0x55febc850530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc850ba0 0x55febc853030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55febc7b0530 con 0x55febc850ba0 2024-04-22T01:14:14.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55febc758830 con 0x55febc84b040 2024-04-22T01:14:14.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f6689898640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f66700a2c20 0x7f66700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f668dfc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55febc7ac4f0 con 0x55febc84ba30 2024-04-22T01:14:14.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.420+0000 7f667a7fc640 1 -- 172.21.15.38:0/2377066479 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f66740f1080 con 0x7f6684101870 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 0x55febc850530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 0x55febc850530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51034/0 (socket says 172.21.15.38:51034) 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 -- 172.21.15.38:0/2163163148 learned_addr learned my addr 172.21.15.38:0/2163163148 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6689097640 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc84b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6689898640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f66700a2c20 0x7f66700a50e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f667c005fd0 tx=0x7f667c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 -- 172.21.15.38:0/2163163148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 msgr2=0x55febc84b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc84b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 -- 172.21.15.38:0/2163163148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc850ba0 msgr2=0x55febc853030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T01:14:14.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc850ba0 0x55febc853030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 -- 172.21.15.38:0/2163163148 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55febc77c2c0 con 0x55febc84ba30 2024-04-22T01:14:14.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f6688896640 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 0x55febc850530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f66740b9920 tx=0x7f6674005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e1f730e8d2e4af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.421+0000 7f666f7fe640 1 -- 172.21.15.38:0/2163163148 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667404b040 con 0x55febc84ba30 2024-04-22T01:14:14.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.422+0000 7f666f7fe640 1 -- 172.21.15.38:0/2163163148 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66740bdbc0 con 0x55febc84ba30 2024-04-22T01:14:14.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.422+0000 7f666f7fe640 1 -- 172.21.15.38:0/2163163148 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66740b9b30 con 0x55febc84ba30 2024-04-22T01:14:14.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.422+0000 7f668dfc0740 1 -- 172.21.15.38:0/2163163148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 msgr2=0x55febc850530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.422+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 0x55febc850530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f66740b9920 tx=0x7f6674005e50 comp rx=0 tx=0).stop 2024-04-22T01:14:14.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 -- 172.21.15.38:0/2163163148 shutdown_connections 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc850ba0 0x55febc853030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84ba30 0x55febc850530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2163163148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc84b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 -- 172.21.15.38:0/2163163148 >> 172.21.15.38:0/2163163148 conn(0x55febc8465b0 msgr2=0x55febc8469b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 -- 172.21.15.38:0/2163163148 shutdown_connections 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 -- 172.21.15.38:0/2163163148 wait complete. 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 -- start start 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.423+0000 7f668dfc0740 1 Processor -- start 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc85d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc85dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc850ba0 0x55febc864d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55febc85c120 con 0x55febc84ba30 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55febc76bf10 con 0x55febc84b040 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55febc85c2a0 con 0x55febc850ba0 2024-04-22T01:14:14.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6688896640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc85dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc85d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689898640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc850ba0 0x55febc864d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc85d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60614/0 (socket says 172.21.15.38:60614) 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 -- 172.21.15.38:0/4066800822 learned_addr learned my addr 172.21.15.38:0/4066800822 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f668dfc0740 10 client.0 _ll_register_callbacks cb 0x55febc4c3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 -- 172.21.15.38:0/4066800822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc850ba0 msgr2=0x55febc864d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc850ba0 0x55febc864d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 -- 172.21.15.38:0/4066800822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 msgr2=0x55febc85dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc85dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6689097640 1 -- 172.21.15.38:0/4066800822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55febc73dc80 con 0x55febc84b040 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.424+0000 7f6688896640 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc85dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f6689097640 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc85d7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f66800126e0 tx=0x7f6680012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9592997740eae803 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f666cff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f666cff9640 1 -- 172.21.15.38:0/4066800822 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66800121d0 con 0x55febc84b040 2024-04-22T01:14:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66740bdbc0 con 0x55febc84b040 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f666cff9640 1 -- 172.21.15.38:0/4066800822 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6680004050 con 0x55febc84b040 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f666cff9640 1 -- 172.21.15.38:0/4066800822 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6680025050 con 0x55febc84b040 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 10 client.25544 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55febc4c43f0 con 0x55febc84b040 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f666cff9640 1 -- 172.21.15.38:0/4066800822 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66740bdbc0 con 0x55febc84b040 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f666cff9640 1 -- 172.21.15.38:0/4066800822 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f66800038c0 con 0x55febc84b040 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 10 client.25544 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 20 client.25544 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc8694f0 0x55febc86b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 4 client.25544 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.425+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55febc7bda70 con 0x55febc8694f0 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.426+0000 7f6688896640 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc8694f0 0x55febc86b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.426+0000 7f666cff9640 10 client.25544 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:14.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.426+0000 7f6688896640 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc8694f0 0x55febc86b990 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92e3a9877591b9e2 server_cookie=2855cdb39110fd0f in_seq=0 out_seq=0 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f666cff9640 1 -- 172.21.15.38:0/4066800822 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55febc7bda70 con 0x55febc8694f0 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f666cff9640 10 client.25544 handle_command_reply: tid=0 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 client.25544 shutdown 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 msgr2=0x55febc85d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc85d7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f66800126e0 tx=0x7f6680012ba0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 shutdown_connections 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc8694f0 0x55febc86b990 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc850ba0 0x55febc864d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f666cff9640 0 client.25544 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc85dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 --2- 172.21.15.38:0/4066800822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84b040 0x55febc85d7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 >> 172.21.15.38:0/4066800822 conn(0x55febc8465b0 msgr2=0x55febc845850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 shutdown_connections 2024-04-22T01:14:14.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 1 -- 172.21.15.38:0/4066800822 wait complete. 2024-04-22T01:14:14.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.427+0000 7f668dfc0740 20 client.25544 trim_cache size 0 max 16384 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.445+0000 7f668dfc0740 1 Processor -- start 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.445+0000 7f668dfc0740 1 -- start start 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 0x55febc84c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc864ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc84ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f668dfc0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66841a8380 con 0x55febc84ba30 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f668dfc0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f66841a9ec0 con 0x55febc8465b0 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f668dfc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f66841a8a50 con 0x55febc84b040 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 0x55febc84c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689898640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc84ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6688896640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc864ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 0x55febc84c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60618/0 (socket says 172.21.15.38:60618) 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 -- 172.21.15.38:0/2884225388 learned_addr learned my addr 172.21.15.38:0/2884225388 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 -- 172.21.15.38:0/2884225388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 msgr2=0x55febc864ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc864ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 -- 172.21.15.38:0/2884225388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 msgr2=0x55febc84ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc84ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 -- 172.21.15.38:0/2884225388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55febc58e7d0 con 0x55febc8465b0 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689898640 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc84ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.446+0000 7f6689097640 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 0x55febc84c4d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6680009f00 tx=0x7f6680009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9edcba2cdf154681 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.447+0000 7f666effd640 1 -- 172.21.15.38:0/2884225388 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668002e070 con 0x55febc8465b0 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.447+0000 7f666effd640 1 -- 172.21.15.38:0/2884225388 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6680012250 con 0x55febc8465b0 2024-04-22T01:14:14.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.447+0000 7f666effd640 1 -- 172.21.15.38:0/2884225388 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680005560 con 0x55febc8465b0 2024-04-22T01:14:14.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.447+0000 7f668dfc0740 1 -- 172.21.15.38:0/2884225388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 msgr2=0x55febc84c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.447+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 0x55febc84c4d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6680009f00 tx=0x7f6680009640 comp rx=0 tx=0).stop 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 -- 172.21.15.38:0/2884225388 shutdown_connections 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc84ba30 0x55febc84ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc864ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2884225388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc8465b0 0x55febc84c4d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 -- 172.21.15.38:0/2884225388 >> 172.21.15.38:0/2884225388 conn(0x55febc8506a0 msgr2=0x55febc84a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 -- 172.21.15.38:0/2884225388 shutdown_connections 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 -- 172.21.15.38:0/2884225388 wait complete. 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 -- start start 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.448+0000 7f668dfc0740 1 Processor -- start 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc8465b0 0x55febc861900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc861e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 0x55febc8cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66841a97f0 con 0x55febc8465b0 2024-04-22T01:14:14.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f66841a9120 con 0x55febc84ba30 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f66841a6340 con 0x55febc84b040 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f668dfc0740 10 client.0 _ll_register_callbacks cb 0x7f6680009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 0x55febc8cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689097640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc8465b0 0x55febc861900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 0x55febc8cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60630/0 (socket says 172.21.15.38:60630) 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 -- 172.21.15.38:0/2826128766 learned_addr learned my addr 172.21.15.38:0/2826128766 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6688896640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc861e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 -- 172.21.15.38:0/2826128766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 msgr2=0x55febc861e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc861e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 -- 172.21.15.38:0/2826128766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc8465b0 msgr2=0x55febc861900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc8465b0 0x55febc861900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 -- 172.21.15.38:0/2826128766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55febc864f10 con 0x55febc84ba30 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689097640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc8465b0 0x55febc861900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:14.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f6689898640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 0x55febc8cdac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f667c011e70 tx=0x7f667c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=776bdb05b643247a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.449+0000 7f666cff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f666cff9640 1 -- 172.21.15.38:0/2826128766 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c003c40 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f666cff9640 1 -- 172.21.15.38:0/2826128766 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f667c003de0 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6680012250 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f666cff9640 1 -- 172.21.15.38:0/2826128766 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f667c004050 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 10 client.25550 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55febc8cf060 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f666cff9640 1 -- 172.21.15.38:0/2826128766 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f667c0046d0 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f666cff9640 1 -- 172.21.15.38:0/2826128766 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f667c029b80 con 0x55febc84ba30 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 10 client.25550 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 10 client.25550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 20 client.25550 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc850ba0 0x55febc8d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:14.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 4 client.25550 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.450+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6680012250 con 0x55febc850ba0 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.451+0000 7f6689097640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc850ba0 0x55febc8d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.451+0000 7f666cff9640 10 client.25550 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.451+0000 7f6689097640 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc850ba0 0x55febc8d1af0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ea8d7c8a354725 server_cookie=f1725f6451e3ca5a in_seq=0 out_seq=0 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.451+0000 7f666cff9640 1 -- 172.21.15.38:0/2826128766 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6680012250 con 0x55febc850ba0 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.451+0000 7f666cff9640 10 client.25550 handle_command_reply: tid=0 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.451+0000 7f668dfc0740 1 client.25550 shutdown 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 msgr2=0x55febc8cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 0x55febc8cdac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f667c011e70 tx=0x7f667c004a40 comp rx=0 tx=0).stop 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 shutdown_connections 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55febc850ba0 0x55febc8d1af0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55febc84ba30 0x55febc8cdac0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55febc84b040 0x55febc861e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 --2- 172.21.15.38:0/2826128766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55febc8465b0 0x55febc861900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 >> 172.21.15.38:0/2826128766 conn(0x55febc8506a0 msgr2=0x55febc844fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f666cff9640 0 client.25550 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:14.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 shutdown_connections 2024-04-22T01:14:14.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 1 -- 172.21.15.38:0/2826128766 wait complete. 2024-04-22T01:14:14.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.452+0000 7f668dfc0740 20 client.25550 trim_cache size 0 max 16384 2024-04-22T01:14:14.457 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f66700a2c20 msgr2=0x7f66700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f66700a2c20 0x7f66700a50e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f667c005fd0 tx=0x7f667c009500 comp rx=0 tx=0).stop 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 msgr2=0x7f668419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f668419c800 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6674009230 tx=0x7f667402c650 comp rx=0 tx=0).stop 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 shutdown_connections 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f66700a2c20 0x7f66700a50e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6684102230 0x7f668419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6684101870 0x7f668419c800 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 --2- 172.21.15.38:0/2377066479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6684100680 0x7f6684069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 >> 172.21.15.38:0/2377066479 conn(0x7f66840fbbf0 msgr2=0x7f6684104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:14.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 shutdown_connections 2024-04-22T01:14:14.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:14.453+0000 7f668b2fb640 1 -- 172.21.15.38:0/2377066479 wait complete. 2024-04-22T01:14:14.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:14 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3282117104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:14:14.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:14 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3282117104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:14:14.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:14 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3282117104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:14:15.468 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T01:14:15.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:15.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 -- 172.21.15.38:0/600343959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f0102980 msgr2=0x7f35f0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 --2- 172.21.15.38:0/600343959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f0102980 0x7f35f0102e00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009900 tx=0x7f35e402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 -- 172.21.15.38:0/600343959 shutdown_connections 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 --2- 172.21.15.38:0/600343959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35f0103340 0x7f35f0109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 --2- 172.21.15.38:0/600343959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f0102980 0x7f35f0102e00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 --2- 172.21.15.38:0/600343959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 0x7f35f0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 -- 172.21.15.38:0/600343959 >> 172.21.15.38:0/600343959 conn(0x7f35f00fcd20 msgr2=0x7f35f00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 -- 172.21.15.38:0/600343959 shutdown_connections 2024-04-22T01:14:15.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.635+0000 7f35f644d640 1 -- 172.21.15.38:0/600343959 wait complete. 2024-04-22T01:14:15.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 Processor -- start 2024-04-22T01:14:15.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 -- start start 2024-04-22T01:14:15.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 0x7f35f0197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35f0103340 0x7f35f0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 0x7f35f019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35f010d460 con 0x7f35f019d270 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f35f010d2e0 con 0x7f35f0103340 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f644d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35f010d5e0 con 0x7f35f0101790 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.636+0000 7f35f49ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 0x7f35f019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 0x7f35f019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48668/0 (socket says 172.21.15.38:48668) 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 -- 172.21.15.38:0/3892167009 learned_addr learned my addr 172.21.15.38:0/3892167009 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35f0103340 0x7f35f0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35effff640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 0x7f35f0197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 -- 172.21.15.38:0/3892167009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 msgr2=0x7f35f0197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 0x7f35f0197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 -- 172.21.15.38:0/3892167009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35f0103340 msgr2=0x7f35f0198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35f0103340 0x7f35f0198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 -- 172.21.15.38:0/3892167009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f019fd00 con 0x7f35f019d270 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35effff640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 0x7f35f0197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f49ea640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 0x7f35f019f660 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009fd0 tx=0x7f35e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35ed7fa640 1 -- 172.21.15.38:0/3892167009 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8017070 con 0x7f35f019d270 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35ed7fa640 1 -- 172.21.15.38:0/3892167009 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e8003e50 con 0x7f35f019d270 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35ed7fa640 1 -- 172.21.15.38:0/3892167009 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e80106c0 con 0x7f35f019d270 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.637+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f019fff0 con 0x7f35f019d270 2024-04-22T01:14:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.638+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f01a0400 con 0x7f35f019d270 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35ed7fa640 1 -- 172.21.15.38:0/3892167009 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f35e8003970 con 0x7f35f019d270 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 Processor -- start 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 -- start start 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783028d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b78302b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b78302e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b782f82170 con 0x55b783026990 2024-04-22T01:14:15.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b782f2a7c0 con 0x55b7830292c0 2024-04-22T01:14:15.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35f9031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b782f7e130 con 0x55b78302bd20 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783028d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783028d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48678/0 (socket says 172.21.15.38:48678) 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35ef7fe640 1 -- 172.21.15.38:0/2162455378 learned_addr learned my addr 172.21.15.38:0/2162455378 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.641+0000 7f35ed7fa640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35d80a2ca0 0x7f35d80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b78302b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35ed7fa640 1 -- 172.21.15.38:0/3892167009 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f35e80cf340 con 0x7f35f019d270 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b78302e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35f49ea640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35d80a2ca0 0x7f35d80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 -- 172.21.15.38:0/2162455378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 msgr2=0x55b78302e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b78302e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 -- 172.21.15.38:0/2162455378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 msgr2=0x55b783028d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783028d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 -- 172.21.15.38:0/2162455378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b782f3be60 con 0x55b7830292c0 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783028d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35effff640 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b78302b6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35f01027e0 tx=0x7f35e00117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e8be27869538151 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35f49ea640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35d80a2ca0 0x7f35d80a5160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f35e80cc950 tx=0x7f35e800b910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35d67fc640 1 -- 172.21.15.38:0/2162455378 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0012450 con 0x55b7830292c0 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35d67fc640 1 -- 172.21.15.38:0/2162455378 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e0012ad0 con 0x55b7830292c0 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.642+0000 7f35d67fc640 1 -- 172.21.15.38:0/2162455378 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e000fbd0 con 0x55b7830292c0 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 -- 172.21.15.38:0/2162455378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 msgr2=0x55b78302b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b78302b6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35f01027e0 tx=0x7f35e00117b0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 -- 172.21.15.38:0/2162455378 shutdown_connections 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b78302e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b78302b6b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 --2- 172.21.15.38:0/2162455378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783028d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.643+0000 7f35f9031740 1 -- 172.21.15.38:0/2162455378 >> 172.21.15.38:0/2162455378 conn(0x55b782fe54f0 msgr2=0x55b783019290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.644+0000 7f35f9031740 1 -- 172.21.15.38:0/2162455378 shutdown_connections 2024-04-22T01:14:15.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.644+0000 7f35f9031740 1 -- 172.21.15.38:0/2162455378 wait complete. 2024-04-22T01:14:15.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.644+0000 7f35f9031740 1 -- start start 2024-04-22T01:14:15.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.644+0000 7f35f9031740 1 Processor -- start 2024-04-22T01:14:15.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b783026990 0x55b783030710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783030c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 0x55b783037c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b78302ecb0 con 0x55b78302bd20 2024-04-22T01:14:15.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b782f3dd40 con 0x55b7830292c0 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b78302ee30 con 0x55b783026990 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 0x55b783037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 0x55b783037c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48692/0 (socket says 172.21.15.38:48692) 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35ef7fe640 1 -- 172.21.15.38:0/2297848618 learned_addr learned my addr 172.21.15.38:0/2297848618 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783030c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35f9031740 10 client.0 _ll_register_callbacks cb 0x55b782f79820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b783026990 0x55b783030710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 -- 172.21.15.38:0/2297848618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b783026990 msgr2=0x55b783030710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b783026990 0x55b783030710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 -- 172.21.15.38:0/2297848618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 msgr2=0x55b783037c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 0x55b783037c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 -- 172.21.15.38:0/2297848618 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b782d5e7e0 con 0x55b7830292c0 2024-04-22T01:14:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 0x55b783037c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35effff640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783030c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f35e0011cb0 tx=0x7f35e0009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4daa34d48c3d5930 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.645+0000 7f35cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2297848618 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0029030 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2297848618 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f35e0020040 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35e0012ad0 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2297848618 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35e0033050 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 10 client.25562 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b782dbef70 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2297848618 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e0012660 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2297848618 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f35e000fbf0 con 0x55b7830292c0 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 10 client.25562 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 10 client.25562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 20 client.25562 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:15.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b78303e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 4 client.25562 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.646+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35e0012ad0 con 0x55b78303c530 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.647+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b78303e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.647+0000 7f35cf7fe640 10 client.25562 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.647+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b78303e950 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53a4b21374ea345 server_cookie=8a35c46ff71271b0 in_seq=0 out_seq=0 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2297848618 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f35e0012ad0 con 0x55b78303c530 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35cf7fe640 10 client.25562 handle_command_reply: tid=0 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35f9031740 1 client.25562 shutdown 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 msgr2=0x55b783030c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35f9031740 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783030c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f35e0011cb0 tx=0x7f35e0009ef0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 shutdown_connections 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.648+0000 7f35f9031740 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b78303e950 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b78302bd20 0x55b783037c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35cf7fe640 0 client.25562 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783030c50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 1 --2- 172.21.15.38:0/2297848618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b783026990 0x55b783030710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 >> 172.21.15.38:0/2297848618 conn(0x55b782fe54f0 msgr2=0x55b783026210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 shutdown_connections 2024-04-22T01:14:15.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 1 -- 172.21.15.38:0/2297848618 wait complete. 2024-04-22T01:14:15.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.649+0000 7f35f9031740 20 client.25562 trim_cache size 0 max 16384 2024-04-22T01:14:15.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 Processor -- start 2024-04-22T01:14:15.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 -- start start 2024-04-22T01:14:15.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783027ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b7830281e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b783037be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a88f0 con 0x55b783026990 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01aa430 con 0x55b7830292c0 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35f9031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a8fc0 con 0x55b78302bd20 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783027ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783027ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48706/0 (socket says 172.21.15.38:48706) 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35ef7fe640 1 -- 172.21.15.38:0/3968745598 learned_addr learned my addr 172.21.15.38:0/3968745598 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.685+0000 7f35effff640 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b7830281e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35ef7fe640 1 -- 172.21.15.38:0/3968745598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 msgr2=0x55b783037be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b783037be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35ef7fe640 1 -- 172.21.15.38:0/3968745598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 msgr2=0x55b7830281e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b7830281e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35ef7fe640 1 -- 172.21.15.38:0/3968745598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b782c56170 con 0x55b783026990 2024-04-22T01:14:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783027ca0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009230 tx=0x7f35e4045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe181932c7e03884 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35d5ffb640 1 -- 172.21.15.38:0/3968745598 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e4004160 con 0x55b783026990 2024-04-22T01:14:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35d5ffb640 1 -- 172.21.15.38:0/3968745598 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e404f8e0 con 0x55b783026990 2024-04-22T01:14:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.686+0000 7f35d5ffb640 1 -- 172.21.15.38:0/3968745598 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e40046c0 con 0x55b783026990 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 -- 172.21.15.38:0/3968745598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 msgr2=0x55b783027ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783027ca0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009230 tx=0x7f35e4045040 comp rx=0 tx=0).stop 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 -- 172.21.15.38:0/3968745598 shutdown_connections 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b783037be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b7830281e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 --2- 172.21.15.38:0/3968745598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783027ca0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 -- 172.21.15.38:0/3968745598 >> 172.21.15.38:0/3968745598 conn(0x55b782fe54f0 msgr2=0x55b78302f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 -- 172.21.15.38:0/3968745598 shutdown_connections 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 -- 172.21.15.38:0/3968745598 wait complete. 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.687+0000 7f35f9031740 1 -- start start 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 Processor -- start 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783034740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783034c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b7830a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a9d60 con 0x55b783026990 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a9690 con 0x55b7830292c0 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a68b0 con 0x55b78302bd20 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35f9031740 10 client.0 _ll_register_callbacks cb 0x7f35e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783034740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783034740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48716/0 (socket says 172.21.15.38:48716) 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 -- 172.21.15.38:0/2715468445 learned_addr learned my addr 172.21.15.38:0/2715468445 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35effff640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783034c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b7830a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 -- 172.21.15.38:0/2715468445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 msgr2=0x55b7830a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b7830a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 -- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 msgr2=0x55b783034c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783034c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35ef7fe640 1 -- 172.21.15.38:0/2715468445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b783037d50 con 0x55b783026990 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.688+0000 7f35effff640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783034c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35ef7fe640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783034740 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f35e402c240 tx=0x7f35e402df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8543b213a9b3c860 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2715468445 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e40035f0 con 0x55b783026990 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35e404f8e0 con 0x55b783026990 2024-04-22T01:14:15.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2715468445 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f35e4003790 con 0x55b783026990 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2715468445 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35e4003990 con 0x55b783026990 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 10 client.16029 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b7830a1fc0 con 0x55b783026990 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2715468445 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e404f8e0 con 0x55b783026990 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2715468445 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f35e4042a20 con 0x55b783026990 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 10 client.16029 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 20 client.16029 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.689+0000 7f35f9031740 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b7830a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.690+0000 7f35f9031740 4 client.16029 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.690+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b782f56450 con 0x55b78303c530 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.690+0000 7f35effff640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b7830a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.691+0000 7f35effff640 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b7830a4900 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a20054d77f858028 server_cookie=6e9046bcc9e8e34a in_seq=0 out_seq=0 2024-04-22T01:14:15.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.691+0000 7f35cf7fe640 10 client.16029 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.692+0000 7f35cf7fe640 1 -- 172.21.15.38:0/2715468445 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b782f56450 con 0x55b78303c530 2024-04-22T01:14:15.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35cf7fe640 10 client.16029 handle_command_reply: tid=0 2024-04-22T01:14:15.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 client.16029 shutdown 2024-04-22T01:14:15.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 msgr2=0x55b783034740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783034740 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f35e402c240 tx=0x7f35e402df90 comp rx=0 tx=0).stop 2024-04-22T01:14:15.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 shutdown_connections 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b78303c530 0x55b7830a4900 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b78302bd20 0x55b7830a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35cf7fe640 0 client.16029 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b7830292c0 0x55b783034c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 --2- 172.21.15.38:0/2715468445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b783026990 0x55b783034740 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 >> 172.21.15.38:0/2715468445 conn(0x55b782fe54f0 msgr2=0x55b7830305e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 shutdown_connections 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 1 -- 172.21.15.38:0/2715468445 wait complete. 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.693+0000 7f35f9031740 20 client.16029 trim_cache size 0 max 16384 2024-04-22T01:14:15.696 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:14:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T01:14:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "42277bec-5ea9-431d-ab90-121efb2ff754", 2024-04-22T01:14:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T01:14:15.697 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:14:15.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35d80a2ca0 msgr2=0x7f35d80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35d80a2ca0 0x7f35d80a5160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f35e80cc950 tx=0x7f35e800b910 comp rx=0 tx=0).stop 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 msgr2=0x7f35f019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 0x7f35f019f660 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009fd0 tx=0x7f35e800b260 comp rx=0 tx=0).stop 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 shutdown_connections 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f35d80a2ca0 0x7f35d80a5160 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35f019d270 0x7f35f019f660 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35f0103340 0x7f35f0198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 --2- 172.21.15.38:0/3892167009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f0101790 0x7f35f0197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 >> 172.21.15.38:0/3892167009 conn(0x7f35f00fcd20 msgr2=0x7f35f0105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 shutdown_connections 2024-04-22T01:14:15.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.694+0000 7f35f644d640 1 -- 172.21.15.38:0/3892167009 wait complete. 2024-04-22T01:14:15.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:15 smithi038 ceph-mon[28043]: pgmap v412: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:15.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:14:15.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:14:15.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:15 smithi203 ceph-mon[31825]: pgmap v412: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.864+0000 7f986f7fe640 1 -- 172.21.15.38:0/3973918787 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986402cc80 con 0x7f98701006a0 2024-04-22T01:14:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 -- 172.21.15.38:0/3973918787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98701006a0 msgr2=0x7f9870100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3973918787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98701006a0 0x7f9870100ac0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9864009900 tx=0x7f986402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 -- 172.21.15.38:0/3973918787 shutdown_connections 2024-04-22T01:14:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3973918787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9870102250 0x7f9870108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3973918787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9870101890 0x7f9870101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3973918787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98701006a0 0x7f9870100ac0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 -- 172.21.15.38:0/3973918787 >> 172.21.15.38:0/3973918787 conn(0x7f98700fbc10 msgr2=0x7f98700fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 -- 172.21.15.38:0/3973918787 shutdown_connections 2024-04-22T01:14:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.865+0000 7f9876dc9640 1 -- 172.21.15.38:0/3973918787 wait complete. 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 Processor -- start 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 -- start start 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98701006a0 0x7f98701957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 0x7f9870195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9870102250 0x7f987019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f987010b270 con 0x7f9870101890 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f987010b0f0 con 0x7f9870102250 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9876dc9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f987010b3f0 con 0x7f98701006a0 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 0x7f9870195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 0x7f9870195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48732/0 (socket says 172.21.15.38:48732) 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 -- 172.21.15.38:0/3516323749 learned_addr learned my addr 172.21.15.38:0/3516323749 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9875366640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9870102250 0x7f987019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f9874b65640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98701006a0 0x7f98701957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 -- 172.21.15.38:0/3516323749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98701006a0 msgr2=0x7f98701957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98701006a0 0x7f98701957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 -- 172.21.15.38:0/3516323749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9870102250 msgr2=0x7f987019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9870102250 0x7f987019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.866+0000 7f986ffff640 1 -- 172.21.15.38:0/3516323749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987019d410 con 0x7f9870101890 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.867+0000 7f986ffff640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 0x7f9870195d30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f985c009a60 tx=0x7f985c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.867+0000 7f986dffb640 1 -- 172.21.15.38:0/3516323749 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c007d30 con 0x7f9870101890 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.867+0000 7f986dffb640 1 -- 172.21.15.38:0/3516323749 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f985c01c070 con 0x7f9870101890 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.867+0000 7f986dffb640 1 -- 172.21.15.38:0/3516323749 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c0166b0 con 0x7f9870101890 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.867+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987019d700 con 0x7f9870101890 2024-04-22T01:14:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.867+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987019dc00 con 0x7f9870101890 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f986dffb640 1 -- 172.21.15.38:0/3516323749 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f985c017a60 con 0x7f9870101890 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 Processor -- start 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 -- start start 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b47d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 0x556b1b482220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b482890 0x556b1b484d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556b1b3e1b70 con 0x556b1b47d720 2024-04-22T01:14:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556b1b38a470 con 0x556b1b482890 2024-04-22T01:14:15.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f98799b9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b1b3ddb30 con 0x556b1b47cd30 2024-04-22T01:14:15.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f8666cb5640 1 -- 172.21.15.38:0/2688867237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660101870 msgr2=0x7f8660101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f8666cb5640 1 --2- 172.21.15.38:0/2688867237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660101870 0x7f8660101cf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f864c009860 tx=0x7f864c02bb70 comp rx=0 tx=0).stop 2024-04-22T01:14:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f9874b65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 0x556b1b482220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f986ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b482890 0x556b1b484d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f9874b65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 0x556b1b482220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48754/0 (socket says 172.21.15.38:48754) 2024-04-22T01:14:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f9874b65640 1 -- 172.21.15.38:0/2821714100 learned_addr learned my addr 172.21.15.38:0/2821714100 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f9875366640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b47d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.871+0000 7f986dffb640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98600a2d50 0x7f98600a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 -- 172.21.15.38:0/2688867237 shutdown_connections 2024-04-22T01:14:15.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 --2- 172.21.15.38:0/2688867237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660102230 0x7f8660108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 --2- 172.21.15.38:0/2688867237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660101870 0x7f8660101cf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 --2- 172.21.15.38:0/2688867237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8660100680 0x7f8660100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9874b65640 1 -- 172.21.15.38:0/2821714100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 msgr2=0x556b1b47d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9874b65640 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b47d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9874b65640 1 -- 172.21.15.38:0/2821714100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b482890 msgr2=0x556b1b484d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9874b65640 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b482890 0x556b1b484d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9874b65640 1 -- 172.21.15.38:0/2821714100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b1b0f6210 con 0x556b1b47d720 2024-04-22T01:14:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9875366640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98600a2d50 0x7f98600a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 -- 172.21.15.38:0/2688867237 >> 172.21.15.38:0/2688867237 conn(0x7f86600fbbf0 msgr2=0x7f86600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 -- 172.21.15.38:0/2688867237 shutdown_connections 2024-04-22T01:14:15.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f8666cb5640 1 -- 172.21.15.38:0/2688867237 wait complete. 2024-04-22T01:14:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.872+0000 7f9874b65640 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 0x556b1b482220 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f986402c8e0 tx=0x7f9864009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed18cc3a39d6b1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f986dffb640 1 -- 172.21.15.38:0/3516323749 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f985c0d1130 con 0x7f9870101890 2024-04-22T01:14:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f985affd640 1 -- 172.21.15.38:0/2821714100 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864044020 con 0x556b1b47d720 2024-04-22T01:14:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f985affd640 1 -- 172.21.15.38:0/2821714100 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986402cb80 con 0x556b1b47d720 2024-04-22T01:14:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f9875366640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98600a2d50 0x7f98600a5210 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9868005fd0 tx=0x7f98680099a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f985affd640 1 -- 172.21.15.38:0/2821714100 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864004920 con 0x556b1b47d720 2024-04-22T01:14:15.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f8666cb5640 1 Processor -- start 2024-04-22T01:14:15.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.873+0000 7f8666cb5640 1 -- start start 2024-04-22T01:14:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 -- 172.21.15.38:0/2821714100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 msgr2=0x556b1b482220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 0x556b1b482220 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f986402c8e0 tx=0x7f9864009560 comp rx=0 tx=0).stop 2024-04-22T01:14:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 -- 172.21.15.38:0/2821714100 shutdown_connections 2024-04-22T01:14:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b482890 0x556b1b484d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8666cb5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660100680 0x7f8660199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b47d720 0x556b1b482220 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8666cb5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 0x7f866019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 --2- 172.21.15.38:0/2821714100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b47d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 -- 172.21.15.38:0/2821714100 >> 172.21.15.38:0/2821714100 conn(0x556b1b4782a0 msgr2=0x556b1b4786a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f865ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 0x7f866019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f865ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 0x7f866019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48758/0 (socket says 172.21.15.38:48758) 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8666cb5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8660102230 0x7f86601a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f865ffff640 1 -- 172.21.15.38:0/56437537 learned_addr learned my addr 172.21.15.38:0/56437537 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f866010c2e0 con 0x7f8660101870 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f866010c160 con 0x7f8660100680 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8664a51640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660100680 0x7f8660199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f866010c460 con 0x7f8660102230 2024-04-22T01:14:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.874+0000 7f98799b9740 1 -- 172.21.15.38:0/2821714100 shutdown_connections 2024-04-22T01:14:15.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8665252640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8660102230 0x7f86601a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f98799b9740 1 -- 172.21.15.38:0/2821714100 wait complete. 2024-04-22T01:14:15.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8664a51640 1 -- 172.21.15.38:0/56437537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8660102230 msgr2=0x7f86601a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8664a51640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8660102230 0x7f86601a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8664a51640 1 -- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 msgr2=0x7f866019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8664a51640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 0x7f866019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8664a51640 1 -- 172.21.15.38:0/56437537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86601a1960 con 0x7f8660100680 2024-04-22T01:14:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f865ffff640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 0x7f866019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f8664a51640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660100680 0x7f8660199d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f865400d6a0 tx=0x7f865400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f98799b9740 1 -- start start 2024-04-22T01:14:15.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f865dffb640 1 -- 172.21.15.38:0/56437537 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654017070 con 0x7f8660100680 2024-04-22T01:14:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f98799b9740 1 Processor -- start 2024-04-22T01:14:15.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.875+0000 7f865dffb640 1 -- 172.21.15.38:0/56437537 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8654009d00 con 0x7f8660100680 2024-04-22T01:14:15.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86601a1c50 con 0x7f8660100680 2024-04-22T01:14:15.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86601a21a0 con 0x7f8660100680 2024-04-22T01:14:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f98799b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b48b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f865dffb640 1 -- 172.21.15.38:0/56437537 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654003c60 con 0x7f8660100680 2024-04-22T01:14:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f98799b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b48b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f98799b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b492a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f986ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b48b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f986ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b48b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60740/0 (socket says 172.21.15.38:60740) 2024-04-22T01:14:15.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.876+0000 7f9875366640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b492a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f98799b9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556b1b489bf0 con 0x556b1b482890 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f98799b9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556b1b39da10 con 0x556b1b47d720 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f98799b9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b1b489d70 con 0x556b1b47cd30 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 -- 172.21.15.38:0/3780366390 learned_addr learned my addr 172.21.15.38:0/3780366390 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f98799b9740 10 client.0 _ll_register_callbacks cb 0x556b1b3d8b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 -- 172.21.15.38:0/3780366390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 msgr2=0x556b1b48b4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b48b4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 -- 172.21.15.38:0/3780366390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 msgr2=0x556b1b492a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b492a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 -- 172.21.15.38:0/3780366390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b1b1c2620 con 0x556b1b47d720 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f9875366640 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b492a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f986ffff640 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b48b9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f985c007650 tx=0x7f985c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2777a1061ffb4984 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.877+0000 7f984ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f984ffff640 1 -- 172.21.15.38:0/3780366390 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c09cdc0 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f984ffff640 1 -- 172.21.15.38:0/3780366390 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f985c00d220 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986402cb80 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f984ffff640 1 -- 172.21.15.38:0/3780366390 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f985c09db60 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f98799b9740 10 client.25586 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b1b495e10 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f984ffff640 1 -- 172.21.15.38:0/3780366390 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986402cb80 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.878+0000 7f984ffff640 1 -- 172.21.15.38:0/3780366390 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f985c09c320 con 0x556b1b47d720 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f98799b9740 10 client.25586 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f98799b9740 10 client.25586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f98799b9740 20 client.25586 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f98799b9740 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4975e0 0x556b1b4999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f98799b9740 4 client.25586 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b1b3ef0b0 con 0x556b1b4975e0 2024-04-22T01:14:15.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f865dffb640 1 -- 172.21.15.38:0/56437537 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137884+0+0 (secure 0 0 0) 0x7f8654009820 con 0x7f8660100680 2024-04-22T01:14:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f9874b65640 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4975e0 0x556b1b4999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f984ffff640 10 client.25586 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.879+0000 7f9874b65640 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4975e0 0x556b1b4999d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6d89c7cb2b3fbe1 server_cookie=278fe832936ba1fa in_seq=0 out_seq=0 2024-04-22T01:14:15.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f865dffb640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86500a2d20 0x7f86500a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 Processor -- start 2024-04-22T01:14:15.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f865dffb640 1 -- 172.21.15.38:0/56437537 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f86540cef90 con 0x7f8660100680 2024-04-22T01:14:15.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 -- start start 2024-04-22T01:14:15.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f865ffff640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86500a2d20 0x7f86500a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f984ffff640 1 -- 172.21.15.38:0/3780366390 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b1b3ef0b0 con 0x556b1b4975e0 2024-04-22T01:14:15.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 0x5645d63ed370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f984ffff640 10 client.25586 handle_command_reply: tid=0 2024-04-22T01:14:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 client.25586 shutdown 2024-04-22T01:14:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ed940 0x5645d63f2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5645d6352560 con 0x5645d63ed940 2024-04-22T01:14:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5645d62fa730 con 0x5645d63ecf50 2024-04-22T01:14:15.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8669979740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645d634e520 con 0x5645d63f2ab0 2024-04-22T01:14:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 msgr2=0x556b1b48b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b48b9f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f985c007650 tx=0x7f985c009f90 comp rx=0 tx=0).stop 2024-04-22T01:14:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8664a51640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 0x5645d63ed370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8664a51640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 0x5645d63ed370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60750/0 (socket says 172.21.15.38:60750) 2024-04-22T01:14:15.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f8664a51640 1 -- 172.21.15.38:0/1986111458 learned_addr learned my addr 172.21.15.38:0/1986111458 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 shutdown_connections 2024-04-22T01:14:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4975e0 0x556b1b4999d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b492a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b48b9f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 --2- 172.21.15.38:0/3780366390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b48b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 >> 172.21.15.38:0/3780366390 conn(0x556b1b4782a0 msgr2=0x556b1b4774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.880+0000 7f984ffff640 0 client.25586 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8665252640 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8664a51640 1 -- 172.21.15.38:0/1986111458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 msgr2=0x5645d63f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 shutdown_connections 2024-04-22T01:14:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f98799b9740 1 -- 172.21.15.38:0/3780366390 wait complete. 2024-04-22T01:14:15.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8664a51640 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8664a51640 1 -- 172.21.15.38:0/1986111458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ed940 msgr2=0x5645d63f2440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T01:14:15.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8664a51640 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ed940 0x5645d63f2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f865ffff640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86500a2d20 0x7f86500a51e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f864c0095f0 tx=0x7f864c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8664a51640 1 -- 172.21.15.38:0/1986111458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d61307d0 con 0x5645d63ecf50 2024-04-22T01:14:15.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8665252640 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:15.896 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f98799b9740 20 client.25586 trim_cache size 0 max 16384 2024-04-22T01:14:15.897 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:15 smithi111 ceph-mon[31057]: pgmap v412: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f8664a51640 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 0x5645d63ed370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f865409b210 tx=0x7f86540993c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6997b2bda65257d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f864affd640 1 -- 172.21.15.38:0/1986111458 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654020040 con 0x5645d63ecf50 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f864affd640 1 -- 172.21.15.38:0/1986111458 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865409f630 con 0x5645d63ecf50 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.881+0000 7f864affd640 1 -- 172.21.15.38:0/1986111458 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540cc780 con 0x5645d63ecf50 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 -- 172.21.15.38:0/1986111458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 msgr2=0x5645d63ed370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 0x5645d63ed370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f865409b210 tx=0x7f86540993c0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 -- 172.21.15.38:0/1986111458 shutdown_connections 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ed940 0x5645d63f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 --2- 172.21.15.38:0/1986111458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ecf50 0x5645d63ed370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 -- 172.21.15.38:0/1986111458 >> 172.21.15.38:0/1986111458 conn(0x5645d63e84c0 msgr2=0x5645d63e88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 -- 172.21.15.38:0/1986111458 shutdown_connections 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.882+0000 7f8669979740 1 -- 172.21.15.38:0/1986111458 wait complete. 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 -- start start 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 Processor -- start 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63f7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d63fea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5645d63f5a00 con 0x5645d63f2ab0 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5645d630de00 con 0x5645d63ed940 2024-04-22T01:14:15.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645d63f5b80 con 0x5645d63ecf50 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.883+0000 7f8669979740 10 client.0 _ll_register_callbacks cb 0x5645d6348eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63f7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63f7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60764/0 (socket says 172.21.15.38:60764) 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 -- 172.21.15.38:0/3492080208 learned_addr learned my addr 172.21.15.38:0/3492080208 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f8665252640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d63fea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f8664a51640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 -- 172.21.15.38:0/3492080208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 msgr2=0x5645d63f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 -- 172.21.15.38:0/3492080208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 msgr2=0x5645d63fea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d63fea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 -- 172.21.15.38:0/3492080208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d60663f0 con 0x5645d63ed940 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f8664a51640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:15.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f865ffff640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63f7a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f864c0349e0 tx=0x7f864c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b0f8df1c824488d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.884+0000 7f863ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f863ffff640 1 -- 172.21.15.38:0/3492080208 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c04f020 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f863ffff640 1 -- 172.21.15.38:0/3492080208 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f864c02caa0 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f865409f630 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f863ffff640 1 -- 172.21.15.38:0/3492080208 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f864c039040 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f8669979740 10 client.25592 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5645d60260f0 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.885+0000 7f863ffff640 1 -- 172.21.15.38:0/3492080208 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865409f630 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.886+0000 7f863ffff640 1 -- 172.21.15.38:0/3492080208 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f864c05c020 con 0x5645d63ed940 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8669979740 10 client.25592 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8669979740 10 client.25592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8669979740 20 client.25592 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8669979740 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d640bbf0 0x5645d640e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8669979740 4 client.25592 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645d635faa0 con 0x5645d640bbf0 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8664a51640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d640bbf0 0x5645d640e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f8664a51640 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d640bbf0 0x5645d640e010 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b8d048700194f4 server_cookie=e3b3a0f1bd4bd602 in_seq=0 out_seq=0 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.887+0000 7f863ffff640 10 client.25592 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f863ffff640 1 -- 172.21.15.38:0/3492080208 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645d635faa0 con 0x5645d640bbf0 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f863ffff640 10 client.25592 handle_command_reply: tid=0 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 client.25592 shutdown 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 msgr2=0x5645d63f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63f7a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f864c0349e0 tx=0x7f864c005f90 comp rx=0 tx=0).stop 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 shutdown_connections 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d640bbf0 0x5645d640e010 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d63fea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63f7a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 --2- 172.21.15.38:0/3492080208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 >> 172.21.15.38:0/3492080208 conn(0x5645d63e84c0 msgr2=0x5645d63e76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f863ffff640 0 client.25592 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 shutdown_connections 2024-04-22T01:14:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 1 -- 172.21.15.38:0/3492080208 wait complete. 2024-04-22T01:14:15.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.888+0000 7f8669979740 20 client.25592 trim_cache size 0 max 16384 2024-04-22T01:14:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 Processor -- start 2024-04-22T01:14:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 -- start start 2024-04-22T01:14:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 0x5645d63f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f864400fcc0 con 0x5645d63ecf50 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8644011800 con 0x5645d63ed940 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8669979740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8644010390 con 0x5645d63f2ab0 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8664a51640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 0x5645d63f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8664a51640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 0x5645d63f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48798/0 (socket says 172.21.15.38:48798) 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8664a51640 1 -- 172.21.15.38:0/778803886 learned_addr learned my addr 172.21.15.38:0/778803886 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f8665252640 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.906+0000 7f865ffff640 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f8664a51640 1 -- 172.21.15.38:0/778803886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 msgr2=0x5645d63e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f8664a51640 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f8664a51640 1 -- 172.21.15.38:0/778803886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 msgr2=0x5645d63fea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f8664a51640 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f8664a51640 1 -- 172.21.15.38:0/778803886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d63e9230 con 0x5645d63ecf50 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f865ffff640 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:15.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.907+0000 7f8664a51640 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 0x5645d63f6ea0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f865409af80 tx=0x7f86540036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f105a88510e28ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f864a7fc640 1 -- 172.21.15.38:0/778803886 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540b3020 con 0x5645d63ecf50 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f864a7fc640 1 -- 172.21.15.38:0/778803886 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865409f630 con 0x5645d63ecf50 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f864a7fc640 1 -- 172.21.15.38:0/778803886 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654020070 con 0x5645d63ecf50 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 -- 172.21.15.38:0/778803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 msgr2=0x5645d63f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 0x5645d63f6ea0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f865409af80 tx=0x7f86540036f0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 -- 172.21.15.38:0/778803886 shutdown_connections 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63f2ab0 0x5645d63e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 --2- 172.21.15.38:0/778803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63ecf50 0x5645d63f6ea0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 -- 172.21.15.38:0/778803886 >> 172.21.15.38:0/778803886 conn(0x5645d63e84c0 msgr2=0x5645d63f6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.908+0000 7f8669979740 1 -- 172.21.15.38:0/778803886 shutdown_connections 2024-04-22T01:14:15.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 -- 172.21.15.38:0/778803886 wait complete. 2024-04-22T01:14:15.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 -- start start 2024-04-22T01:14:15.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 Processor -- start 2024-04-22T01:14:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f98799b9740 1 Processor -- start 2024-04-22T01:14:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f98799b9740 1 -- start start 2024-04-22T01:14:15.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63fb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d6470120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8644011130 con 0x5645d63f2ab0 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8644010a60 con 0x5645d63ed940 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f864400dc80 con 0x5645d63ecf50 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.909+0000 7f8669979740 10 client.0 _ll_register_callbacks cb 0x7f8658009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f8665252640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d6470120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f8664a51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63fb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f8664a51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63fb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51204/0 (socket says 172.21.15.38:51204) 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60784/0 (socket says 172.21.15.38:60784) 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f8664a51640 1 -- 172.21.15.38:0/3547043211 learned_addr learned my addr 172.21.15.38:0/3547043211 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 -- 172.21.15.38:0/3547043211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 msgr2=0x5645d63fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63fb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 -- 172.21.15.38:0/3547043211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 msgr2=0x5645d6470120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d6470120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 -- 172.21.15.38:0/3547043211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d6470660 con 0x5645d63ed940 2024-04-22T01:14:15.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f8664a51640 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:15.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f865ffff640 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fbdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f864c009830 tx=0x7f864c004540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=127fc5e9bb364845 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f863ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f98799b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b499c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f98799b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b4929f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f98799b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b473220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b499c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b499c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51212/0 (socket says 172.21.15.38:51212) 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 -- 172.21.15.38:0/3955637796 learned_addr learned my addr 172.21.15.38:0/3955637796 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98701a8360 con 0x556b1b482890 2024-04-22T01:14:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f98701a9ea0 con 0x556b1b47d720 2024-04-22T01:14:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98701a8a30 con 0x556b1b47cd30 2024-04-22T01:14:15.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f863ffff640 1 -- 172.21.15.38:0/3547043211 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c061020 con 0x5645d63ed940 2024-04-22T01:14:15.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f863ffff640 1 -- 172.21.15.38:0/3547043211 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f864c0054d0 con 0x5645d63ed940 2024-04-22T01:14:15.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.910+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f865409f630 con 0x5645d63ed940 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f863ffff640 1 -- 172.21.15.38:0/3547043211 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f864c06c050 con 0x5645d63ed940 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 10 client.25598 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5645d6471910 con 0x5645d63ed940 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f863ffff640 1 -- 172.21.15.38:0/3547043211 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f864c003db0 con 0x5645d63ed940 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f863ffff640 1 -- 172.21.15.38:0/3547043211 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f864c05c040 con 0x5645d63ed940 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 10 client.25598 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 10 client.25598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 20 client.25598 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d6472340 0x5645d6474760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 4 client.25598 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:14:15.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f865409f630 con 0x5645d6472340 2024-04-22T01:14:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 -- 172.21.15.38:0/3955637796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 msgr2=0x556b1b4929f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T01:14:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b4929f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 -- 172.21.15.38:0/3955637796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 msgr2=0x556b1b473220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b473220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 -- 172.21.15.38:0/3955637796 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b1b479010 con 0x556b1b47cd30 2024-04-22T01:14:15.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f8664a51640 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d6472340 0x5645d6474760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f9874b65640 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b499c10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9864009900 tx=0x7f986402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d8f4bb09f28369c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.911+0000 7f985a7fc640 1 -- 172.21.15.38:0/3955637796 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864044040 con 0x556b1b47cd30 2024-04-22T01:14:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f985a7fc640 1 -- 172.21.15.38:0/3955637796 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986402cb80 con 0x556b1b47cd30 2024-04-22T01:14:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f985a7fc640 1 -- 172.21.15.38:0/3955637796 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98640406d0 con 0x556b1b47cd30 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f863ffff640 10 client.25598 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8664a51640 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d6472340 0x5645d6474760 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=450dac16f504d114 server_cookie=d007dc5ea80d74b1 in_seq=0 out_seq=0 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f863ffff640 1 -- 172.21.15.38:0/3547043211 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f865409f630 con 0x5645d6472340 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f863ffff640 10 client.25598 handle_command_reply: tid=0 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 client.25598 shutdown 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 msgr2=0x5645d63fbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fbdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f864c009830 tx=0x7f864c004540 comp rx=0 tx=0).stop 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 shutdown_connections 2024-04-22T01:14:15.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5645d6472340 0x5645d6474760 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 msgr2=0x556b1b499c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b499c10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9864009900 tx=0x7f986402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:15.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f863ffff640 0 client.25598 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645d63f2ab0 0x5645d6470120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5645d63ed940 0x5645d63fbdb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 --2- 172.21.15.38:0/3547043211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d63ecf50 0x5645d63fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 >> 172.21.15.38:0/3547043211 conn(0x5645d63e84c0 msgr2=0x5645d63eb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 shutdown_connections 2024-04-22T01:14:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b473220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47d720 0x556b1b4929f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 --2- 172.21.15.38:0/3955637796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47cd30 0x556b1b499c10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.912+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 >> 172.21.15.38:0/3955637796 conn(0x556b1b4782a0 msgr2=0x556b1b499500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 shutdown_connections 2024-04-22T01:14:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f98799b9740 1 -- 172.21.15.38:0/3955637796 wait complete. 2024-04-22T01:14:15.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 shutdown_connections 2024-04-22T01:14:15.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8669979740 1 -- 172.21.15.38:0/3547043211 wait complete. 2024-04-22T01:14:15.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8669979740 20 client.25598 trim_cache size 0 max 16384 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: "42277bec-5ea9-431d-ab90-121efb2ff754": { 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "42277bec-5ea9-431d-ab90-121efb2ff754", 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:14:15.922 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:14:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f98799b9740 1 -- start start 2024-04-22T01:14:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f98799b9740 1 Processor -- start 2024-04-22T01:14:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86500a2d20 msgr2=0x7f86500a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86500a2d20 0x7f86500a51e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f864c0095f0 tx=0x7f864c02e040 comp rx=0 tx=0).stop 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660100680 msgr2=0x7f8660199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660100680 0x7f8660199d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f865400d6a0 tx=0x7f865400db60 comp rx=0 tx=0).stop 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 shutdown_connections 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f86500a2d20 0x7f86500a51e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8660102230 0x7f86601a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8660101870 0x7f866019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 --2- 172.21.15.38:0/56437537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8660100680 0x7f8660199d50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f98799b9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 0x556b1b48fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.913+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 >> 172.21.15.38:0/56437537 conn(0x7f86600fbbf0 msgr2=0x7f86600fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 shutdown_connections 2024-04-22T01:14:15.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f98799b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47d720 0x556b1b4901e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f8666cb5640 1 -- 172.21.15.38:0/56437537 wait complete. 2024-04-22T01:14:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 0x556b1b48fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 0x556b1b48fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60804/0 (socket says 172.21.15.38:60804) 2024-04-22T01:14:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 -- 172.21.15.38:0/2670924129 learned_addr learned my addr 172.21.15.38:0/2670924129 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f986ffff640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47d720 0x556b1b4901e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b4911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98701a97d0 con 0x556b1b482890 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f98701a9100 con 0x556b1b47cd30 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98701a6320 con 0x556b1b47d720 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9875366640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b4911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 -- 172.21.15.38:0/2670924129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47d720 msgr2=0x556b1b4901e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47d720 0x556b1b4901e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 -- 172.21.15.38:0/2670924129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 msgr2=0x556b1b4911a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b4911a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.914+0000 7f9874b65640 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b1b491790 con 0x556b1b47cd30 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f9874b65640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 0x556b1b48fca0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f986404b040 tx=0x7f986402c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a0f5b8b844a8ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f984ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f984ffff640 1 -- 172.21.15.38:0/2670924129 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864044070 con 0x556b1b47cd30 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f984ffff640 1 -- 172.21.15.38:0/2670924129 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f9864004260 con 0x556b1b47cd30 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f98799b9740 10 client.0 _ll_register_callbacks cb 0x7f985c00d590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986402cb80 con 0x556b1b47cd30 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f984ffff640 1 -- 172.21.15.38:0/2670924129 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f986405c050 con 0x556b1b47cd30 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f98799b9740 10 client.25601 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.915+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=28}) -- 0x556b1b492750 con 0x556b1b47cd30 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f984ffff640 1 -- 172.21.15.38:0/2670924129 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f986404e170 con 0x556b1b47cd30 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f98799b9740 10 client.25601 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f98799b9740 10 client.25601 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f98799b9740 20 client.25601 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4fda40 0x556b1b4ffe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f98799b9740 4 client.25601 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:14:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f986402cb80 con 0x556b1b4fda40 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.916+0000 7f986ffff640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4fda40 0x556b1b4ffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.917+0000 7f984ffff640 10 client.25601 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.917+0000 7f986ffff640 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4fda40 0x556b1b4ffe30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47640bc054eff553 server_cookie=d70d93c51c8df306 in_seq=0 out_seq=0 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f984ffff640 1 -- 172.21.15.38:0/2670924129 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f986402cb80 con 0x556b1b4fda40 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f984ffff640 10 client.25601 handle_command_reply: tid=0 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 client.25601 shutdown 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 msgr2=0x556b1b48fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 0x556b1b48fca0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f986404b040 tx=0x7f986402c360 comp rx=0 tx=0).stop 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 shutdown_connections 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x556b1b4fda40 0x556b1b4ffe30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b1b482890 0x556b1b4911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b1b47d720 0x556b1b4901e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f984ffff640 0 client.25601 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 --2- 172.21.15.38:0/2670924129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556b1b47cd30 0x556b1b48fca0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 >> 172.21.15.38:0/2670924129 conn(0x556b1b4782a0 msgr2=0x556b1b47b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 shutdown_connections 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 1 -- 172.21.15.38:0/2670924129 wait complete. 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.922+0000 7f98799b9740 20 client.25601 trim_cache size 0 max 16384 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:14:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:14:15.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98600a2d50 msgr2=0x7f98600a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98600a2d50 0x7f98600a5210 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9868005fd0 tx=0x7f98680099a0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 msgr2=0x7f9870195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 0x7f9870195d30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f985c009a60 tx=0x7f985c009f20 comp rx=0 tx=0).stop 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 shutdown_connections 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f98600a2d50 0x7f98600a5210 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9870102250 0x7f987019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9870101890 0x7f9870195d30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 --2- 172.21.15.38:0/3516323749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98701006a0 0x7f98701957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.923+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 >> 172.21.15.38:0/3516323749 conn(0x7f98700fbc10 msgr2=0x7f9870104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.924+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 shutdown_connections 2024-04-22T01:14:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:15.924+0000 7f9876dc9640 1 -- 172.21.15.38:0/3516323749 wait complete. 2024-04-22T01:14:15.935 INFO:tasks.cephfs.filesystem:scrub status for tag:42277bec-5ea9-431d-ab90-121efb2ff754 - {'path': '/', 'tag': '42277bec-5ea9-431d-ab90-121efb2ff754', 'options': 'recursive,force'} 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: scrub queued for path: / 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: scrub summary: active paths [/] 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: pgmap v413: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: scrub summary: idle 2024-04-22T01:14:17.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:17 smithi203 ceph-mon[31825]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: scrub queued for path: / 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: scrub summary: active paths [/] 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: pgmap v413: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: scrub summary: idle 2024-04-22T01:14:17.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:17 smithi111 ceph-mon[31057]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-04-22T01:14:17.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: scrub queued for path: / 2024-04-22T01:14:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: scrub summary: active paths [/] 2024-04-22T01:14:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: pgmap v413: 129 pgs: 129 active+clean; 672 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T01:14:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: scrub summary: idle 2024-04-22T01:14:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:17 smithi038 ceph-mon[28043]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-04-22T01:14:18.669 INFO:tasks.workunit.client.0.smithi038.stderr:Updating files: 96% (12004/12485) Updating files: 97% (12111/12485) Updating files: 98% (12236/12485) Updating files: 99% (12361/12485) Updating files: 100% (12485/12485) Updating files: 100% (12485/12485), done. 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr:Note: switching to '430e09df97c8fc7dc2b2ae424f68ed11366c540f'. 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.351 INFO:tasks.workunit.client.0.smithi038.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: git switch -c 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr:Or undo this operation with: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: git switch - 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T01:14:19.352 INFO:tasks.workunit.client.0.smithi038.stderr:HEAD is now at 430e09df97c Merge pull request #57017 from zdover23/wip-doc-2024-04-21-backport-56904-to-squid 2024-04-22T01:14:19.355 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T01:14:19.392 INFO:tasks.workunit.client.0.smithi038.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T01:14:19.394 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T01:14:19.394 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T01:14:19.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:19 smithi203 ceph-mon[31825]: pgmap v414: 129 pgs: 129 active+clean; 673 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1023 B/s wr, 13 op/s 2024-04-22T01:14:19.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:19 smithi111 ceph-mon[31057]: pgmap v414: 129 pgs: 129 active+clean; 673 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1023 B/s wr, 13 op/s 2024-04-22T01:14:19.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:19 smithi038 ceph-mon[28043]: pgmap v414: 129 pgs: 129 active+clean; 673 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1023 B/s wr, 13 op/s 2024-04-22T01:14:20.042 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T01:14:20.095 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T01:14:20.136 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T01:14:20.137 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T01:14:20.137 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T01:14:20.177 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T01:14:20.181 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:14:20.181 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T01:14:20.238 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-04-22T01:14:20.241 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-04-22T01:14:20.241 DEBUG:teuthology.orchestra.run.smithi038:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-04-22T01:14:20.299 INFO:tasks.workunit.client.0.smithi038.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-22T01:14:20.372 INFO:tasks.workunit.client.0.smithi038.stdout: Iozone: Performance Test of File I/O 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Version $Revision: 3.494 $ 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Compiled for 64 bit mode. 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Build: linux-AMD64 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-22T01:14:20.373 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Run began: Mon Apr 22 01:14:20 2024 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Include close in write timing 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Include fsync in write timing 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: File size set to 1048576 kB 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Record Size 16 kB 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Output is in kBytes/sec 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Time Resolution = 0.000001 seconds. 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Processor cache size set to 1024 kBytes. 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Processor cache line size set to 32 bytes. 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: File stride size set to 17 * record size. 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Throughput test with 1 process 2024-04-22T01:14:20.374 INFO:tasks.workunit.client.0.smithi038.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-04-22T01:14:21.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:21 smithi203 ceph-mon[31825]: pgmap v415: 129 pgs: 129 active+clean; 674 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 initial writers = 725069.44 kB/sec 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 initial writers = 724986.76 kB/sec 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 725069.44 kB/sec 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 725069.44 kB/sec 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 725069.44 kB/sec 2024-04-22T01:14:21.821 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:21.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:21 smithi111 ceph-mon[31057]: pgmap v415: 129 pgs: 129 active+clean; 674 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-22T01:14:21.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:21 smithi038 ceph-mon[28043]: pgmap v415: 129 pgs: 129 active+clean; 674 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-22T01:14:23.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:23 smithi203 ceph-mon[31825]: pgmap v416: 129 pgs: 129 active+clean; 201 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 19 op/s 2024-04-22T01:14:23.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:23 smithi111 ceph-mon[31057]: pgmap v416: 129 pgs: 129 active+clean; 201 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 19 op/s 2024-04-22T01:14:23.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:23 smithi038 ceph-mon[28043]: pgmap v416: 129 pgs: 129 active+clean; 201 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 19 op/s 2024-04-22T01:14:25.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:25 smithi203 ceph-mon[31825]: pgmap v417: 129 pgs: 129 active+clean; 429 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 36 MiB/s wr, 22 op/s 2024-04-22T01:14:25.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:25 smithi111 ceph-mon[31057]: pgmap v417: 129 pgs: 129 active+clean; 429 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 36 MiB/s wr, 22 op/s 2024-04-22T01:14:25.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:25 smithi038 ceph-mon[28043]: pgmap v417: 129 pgs: 129 active+clean; 429 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 36 MiB/s wr, 22 op/s 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 rewriters = 768485.38 kB/sec 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 rewriters = 768355.32 kB/sec 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 768485.38 kB/sec 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 768485.38 kB/sec 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 768485.38 kB/sec 2024-04-22T01:14:27.011 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:27.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:27 smithi203 ceph-mon[31825]: pgmap v418: 129 pgs: 129 active+clean; 861 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 72 MiB/s wr, 31 op/s 2024-04-22T01:14:27.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:27 smithi111 ceph-mon[31057]: pgmap v418: 129 pgs: 129 active+clean; 861 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 72 MiB/s wr, 31 op/s 2024-04-22T01:14:27.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:27 smithi038 ceph-mon[28043]: pgmap v418: 129 pgs: 129 active+clean; 861 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 72 MiB/s wr, 31 op/s 2024-04-22T01:14:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 -- 172.21.15.38:0/3079726508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f841039c0 msgr2=0x7f0f84103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/3079726508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f841039c0 0x7f0f84103de0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009900 tx=0x7f0f7802bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 -- 172.21.15.38:0/3079726508 shutdown_connections 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/3079726508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 0x7f0f8410be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/3079726508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f84104bb0 0x7f0f84105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/3079726508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f841039c0 0x7f0f84103de0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.025+0000 7f0f8b07d640 1 -- 172.21.15.38:0/3079726508 >> 172.21.15.38:0/3079726508 conn(0x7f0f840fef30 msgr2=0x7f0f84101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 -- 172.21.15.38:0/3079726508 shutdown_connections 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 -- 172.21.15.38:0/3079726508 wait complete. 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 Processor -- start 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 -- start start 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f841039c0 0x7f0f84199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 0x7f0f8419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 0x7f0f841a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8410f690 con 0x7f0f84104bb0 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8410f510 con 0x7f0f841039c0 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f8b07d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8410f810 con 0x7f0f84105570 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.026+0000 7f0f83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 0x7f0f8419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 0x7f0f8419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34480/0 (socket says 172.21.15.38:34480) 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 -- 172.21.15.38:0/2850791615 learned_addr learned my addr 172.21.15.38:0/2850791615 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f88e19640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f841039c0 0x7f0f84199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f8961a640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 0x7f0f841a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 -- 172.21.15.38:0/2850791615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 msgr2=0x7f0f841a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 0x7f0f841a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 -- 172.21.15.38:0/2850791615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f841039c0 msgr2=0x7f0f84199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f841039c0 0x7f0f84199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 -- 172.21.15.38:0/2850791615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f841a1850 con 0x7f0f84104bb0 2024-04-22T01:14:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f8961a640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 0x7f0f841a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f83fff640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 0x7f0f8419a180 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000d950 tx=0x7f0f7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f81ffb640 1 -- 172.21.15.38:0/2850791615 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70017070 con 0x7f0f84104bb0 2024-04-22T01:14:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f81ffb640 1 -- 172.21.15.38:0/2850791615 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f70009c30 con 0x7f0f84104bb0 2024-04-22T01:14:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f841a1b40 con 0x7f0f84104bb0 2024-04-22T01:14:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.027+0000 7f0f81ffb640 1 -- 172.21.15.38:0/2850791615 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000f6b0 con 0x7f0f84104bb0 2024-04-22T01:14:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.028+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f841a2060 con 0x7f0f84104bb0 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.029+0000 7f0f81ffb640 1 -- 172.21.15.38:0/2850791615 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0f700109d0 con 0x7f0f84104bb0 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 Processor -- start 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f81ffb640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f740a2c30 0x7f0f740a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 -- start start 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8961a640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f740a2c30 0x7f0f740a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f81ffb640 1 -- 172.21.15.38:0/2850791615 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f0f700cf130 con 0x7f0f84104bb0 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4550d30 0x55b6a4551150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4551720 0x55b6a4556220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 0x55b6a4558d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b6a44b5b70 con 0x55b6a4556890 2024-04-22T01:14:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b6a445e470 con 0x55b6a4551720 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.030+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b6a44b1b30 con 0x55b6a4550d30 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 0x55b6a4558d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 0x55b6a4558d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34496/0 (socket says 172.21.15.38:34496) 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 -- 172.21.15.38:0/1233999282 learned_addr learned my addr 172.21.15.38:0/1233999282 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f88e19640 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4550d30 0x55b6a4551150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f83fff640 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4551720 0x55b6a4556220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f740a2c30 0x7f0f740a50f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c014820 tx=0x7f0f7c014c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 -- 172.21.15.38:0/1233999282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4550d30 msgr2=0x55b6a4551150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4550d30 0x55b6a4551150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 -- 172.21.15.38:0/1233999282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4551720 msgr2=0x55b6a4556220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4551720 0x55b6a4556220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 -- 172.21.15.38:0/1233999282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a41ca210 con 0x55b6a4556890 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f8961a640 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 0x55b6a4558d20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c012760 tx=0x7f0f7c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb48d87b69d99a17 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f6effd640 1 -- 172.21.15.38:0/1233999282 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00eb60 con 0x55b6a4556890 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f6effd640 1 -- 172.21.15.38:0/1233999282 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7c00ed00 con 0x55b6a4556890 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.031+0000 7f0f6effd640 1 -- 172.21.15.38:0/1233999282 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c01f7b0 con 0x55b6a4556890 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1233999282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 msgr2=0x55b6a4558d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 0x55b6a4558d20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c012760 tx=0x7f0f7c01e000 comp rx=0 tx=0).stop 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1233999282 shutdown_connections 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4556890 0x55b6a4558d20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4551720 0x55b6a4556220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1233999282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4550d30 0x55b6a4551150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1233999282 >> 172.21.15.38:0/1233999282 conn(0x55b6a454c2a0 msgr2=0x55b6a454c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1233999282 shutdown_connections 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1233999282 wait complete. 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.032+0000 7f0f8dc63740 1 -- start start 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 Processor -- start 2024-04-22T01:14:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a4562530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b6a45597e0 con 0x55b6a4551720 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4471a10 con 0x55b6a4550d30 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b6a4559960 con 0x55b6a4556890 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8dc63740 10 client.0 _ll_register_callbacks cb 0x55b6a44acb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34502/0 (socket says 172.21.15.38:34502) 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 -- 172.21.15.38:0/662379199 learned_addr learned my addr 172.21.15.38:0/662379199 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f88e19640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f8961a640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a4562530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 -- 172.21.15.38:0/662379199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 msgr2=0x55b6a4562530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a4562530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 -- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 msgr2=0x55b6a455afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.033+0000 7f0f83fff640 1 -- 172.21.15.38:0/662379199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a4296620 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f83fff640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0f70010fd0 tx=0x7f0f70005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fe52a209cd8489 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f63fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f63fff640 1 -- 172.21.15.38:0/662379199 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70020040 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f63fff640 1 -- 172.21.15.38:0/662379199 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f0f700b4560 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7c00ed00 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f63fff640 1 -- 172.21.15.38:0/662379199 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0f7009bcc0 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 10 client.16071 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6a4565960 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f63fff640 1 -- 172.21.15.38:0/662379199 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7c00ed00 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f63fff640 1 -- 172.21.15.38:0/662379199 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0f700b4240 con 0x55b6a4551720 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 10 client.16071 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 20 client.16071 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a4567030 0x55b6a4569460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 4 client.16071 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.034+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6a44c30b0 con 0x55b6a4567030 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.035+0000 7f0f88e19640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a4567030 0x55b6a4569460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.035+0000 7f0f88e19640 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a4567030 0x55b6a4569460 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c99d6519d23ea52a server_cookie=71b2c4df586d03fa in_seq=0 out_seq=0 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.035+0000 7f0f63fff640 10 client.16071 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f63fff640 1 -- 172.21.15.38:0/662379199 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6a44c30b0 con 0x55b6a4567030 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f63fff640 10 client.16071 handle_command_reply: tid=0 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 client.16071 shutdown 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 msgr2=0x55b6a455b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0f70010fd0 tx=0x7f0f70005f90 comp rx=0 tx=0).stop 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 shutdown_connections 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a4567030 0x55b6a4569460 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a4562530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f63fff640 0 client.16071 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/662379199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 >> 172.21.15.38:0/662379199 conn(0x55b6a454c2a0 msgr2=0x55b6a454b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 shutdown_connections 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 1 -- 172.21.15.38:0/662379199 wait complete. 2024-04-22T01:14:29.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.036+0000 7f0f8dc63740 20 client.16071 trim_cache size 0 max 16384 2024-04-22T01:14:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.051+0000 7f0f8dc63740 1 Processor -- start 2024-04-22T01:14:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.051+0000 7f0f8dc63740 1 -- start start 2024-04-22T01:14:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a45624f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841ac6d0 con 0x55b6a4551720 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841ae210 con 0x55b6a4550d30 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841acda0 con 0x55b6a4556890 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34508/0 (socket says 172.21.15.38:34508) 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 -- 172.21.15.38:0/1067747169 learned_addr learned my addr 172.21.15.38:0/1067747169 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f88e19640 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 -- 172.21.15.38:0/1067747169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 msgr2=0x55b6a45624f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a45624f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 -- 172.21.15.38:0/1067747169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 msgr2=0x55b6a455afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 -- 172.21.15.38:0/1067747169 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a4551b00 con 0x55b6a4551720 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f83fff640 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0f700cf3d0 tx=0x7f0f700860b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d461ff840a17d27f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f6e7fc640 1 -- 172.21.15.38:0/1067747169 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70020070 con 0x55b6a4551720 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f6e7fc640 1 -- 172.21.15.38:0/1067747169 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7009b5c0 con 0x55b6a4551720 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.052+0000 7f0f6e7fc640 1 -- 172.21.15.38:0/1067747169 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700afd20 con 0x55b6a4551720 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1067747169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 msgr2=0x55b6a455b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0f700cf3d0 tx=0x7f0f700860b0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1067747169 shutdown_connections 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4556890 0x55b6a45624f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4551720 0x55b6a455b4f0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/1067747169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4550d30 0x55b6a455afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1067747169 >> 172.21.15.38:0/1067747169 conn(0x55b6a454c2a0 msgr2=0x55b6a454c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1067747169 shutdown_connections 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 -- 172.21.15.38:0/1067747169 wait complete. 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 -- start start 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.053+0000 7f0f8dc63740 1 Processor -- start 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 0x55b6a455fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4551720 0x55b6a455ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4556890 0x55b6a4560f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841adb40 con 0x55b6a4550d30 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841ad470 con 0x55b6a4556890 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f841aa690 con 0x55b6a4551720 2024-04-22T01:14:29.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8dc63740 10 client.0 _ll_register_callbacks cb 0x7f0f7009b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 0x55b6a455fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 0x55b6a455fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34514/0 (socket says 172.21.15.38:34514) 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 -- 172.21.15.38:0/2981273449 learned_addr learned my addr 172.21.15.38:0/2981273449 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f8961a640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4556890 0x55b6a4560f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f83fff640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4551720 0x55b6a455ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 -- 172.21.15.38:0/2981273449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4551720 msgr2=0x55b6a455ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4551720 0x55b6a455ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 -- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4556890 msgr2=0x55b6a4560f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4556890 0x55b6a4560f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.054+0000 7f0f88e19640 1 -- 172.21.15.38:0/2981273449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a4561580 con 0x55b6a4550d30 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f88e19640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 0x55b6a455fa40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0f78000c00 tx=0x7f0f7802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bef32ed9a34f184 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f63fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f63fff640 1 -- 172.21.15.38:0/2981273449 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78004250 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7009b5c0 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f63fff640 1 -- 172.21.15.38:0/2981273449 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f0f780043f0 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f63fff640 1 -- 172.21.15.38:0/2981273449 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0f78059050 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 10 client.16077 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6a4562900 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f63fff640 1 -- 172.21.15.38:0/2981273449 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7009b5c0 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f63fff640 1 -- 172.21.15.38:0/2981273449 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0f780046c0 con 0x55b6a4550d30 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 10 client.16077 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 20 client.16077 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a45cde00 0x55b6a45d0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 4 client.16077 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:14:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.055+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6a44bd400 con 0x55b6a45cde00 2024-04-22T01:14:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.056+0000 7f0f83fff640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a45cde00 0x55b6a45d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.056+0000 7f0f83fff640 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a45cde00 0x55b6a45d0220 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87e11c8478ac62b9 server_cookie=5047f65807dd9662 in_seq=0 out_seq=0 2024-04-22T01:14:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.056+0000 7f0f63fff640 10 client.16077 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f63fff640 1 -- 172.21.15.38:0/2981273449 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b6a44bd400 con 0x55b6a45cde00 2024-04-22T01:14:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f63fff640 10 client.16077 handle_command_reply: tid=0 2024-04-22T01:14:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 client.16077 shutdown 2024-04-22T01:14:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 msgr2=0x55b6a455fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 0x55b6a455fa40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0f78000c00 tx=0x7f0f7802cc70 comp rx=0 tx=0).stop 2024-04-22T01:14:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 shutdown_connections 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55b6a45cde00 0x55b6a45d0220 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b6a4556890 0x55b6a4560f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f63fff640 0 client.16077 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6a4551720 0x55b6a455ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 --2- 172.21.15.38:0/2981273449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6a4550d30 0x55b6a455fa40 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.066+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 >> 172.21.15.38:0/2981273449 conn(0x55b6a454c2a0 msgr2=0x55b6a4552a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 shutdown_connections 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8dc63740 1 -- 172.21.15.38:0/2981273449 wait complete. 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8dc63740 20 client.16077 trim_cache size 0 max 16384 2024-04-22T01:14:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:14:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:14:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:14:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f740a2c30 msgr2=0x7f0f740a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f740a2c30 0x7f0f740a50f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c014820 tx=0x7f0f7c014c20 comp rx=0 tx=0).stop 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 msgr2=0x7f0f8419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 0x7f0f8419a180 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000d950 tx=0x7f0f7000de10 comp rx=0 tx=0).stop 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 shutdown_connections 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0f740a2c30 0x7f0f740a50f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f84105570 0x7f0f841a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f84104bb0 0x7f0f8419a180 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 --2- 172.21.15.38:0/2850791615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f841039c0 0x7f0f84199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 >> 172.21.15.38:0/2850791615 conn(0x7f0f840fef30 msgr2=0x7f0f84107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 shutdown_connections 2024-04-22T01:14:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:29.067+0000 7f0f8b07d640 1 -- 172.21.15.38:0/2850791615 wait complete. 2024-04-22T01:14:29.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:29 smithi203 ceph-mon[31825]: pgmap v419: 129 pgs: 129 active+clean; 1.0 GiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 115 MiB/s wr, 45 op/s 2024-04-22T01:14:29.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:29 smithi111 ceph-mon[31057]: pgmap v419: 129 pgs: 129 active+clean; 1.0 GiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 115 MiB/s wr, 45 op/s 2024-04-22T01:14:29.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:29 smithi038 ceph-mon[28043]: pgmap v419: 129 pgs: 129 active+clean; 1.0 GiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 115 MiB/s wr, 45 op/s 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 readers = 6754412.50 kB/sec 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 readers = 6746341.28 kB/sec 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 6754412.50 kB/sec 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 6754412.50 kB/sec 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 6754412.50 kB/sec 2024-04-22T01:14:29.914 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:31.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:31 smithi203 ceph-mon[31825]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 136 MiB/s wr, 45 op/s 2024-04-22T01:14:31.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:31 smithi111 ceph-mon[31057]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 136 MiB/s wr, 45 op/s 2024-04-22T01:14:31.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:31 smithi038 ceph-mon[28043]: pgmap v420: 129 pgs: 129 active+clean; 1.0 GiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 136 MiB/s wr, 45 op/s 2024-04-22T01:14:32.147 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:32.147 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 re-readers = 6744210.00 kB/sec 2024-04-22T01:14:32.148 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 re-readers = 6736452.30 kB/sec 2024-04-22T01:14:32.148 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 6744210.00 kB/sec 2024-04-22T01:14:32.148 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 6744210.00 kB/sec 2024-04-22T01:14:32.148 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 6744210.00 kB/sec 2024-04-22T01:14:32.148 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:33.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:33 smithi203 ceph-mon[31825]: pgmap v421: 129 pgs: 129 active+clean; 1.0 GiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 58 op/s 2024-04-22T01:14:33.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:33 smithi111 ceph-mon[31057]: pgmap v421: 129 pgs: 129 active+clean; 1.0 GiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 58 op/s 2024-04-22T01:14:33.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:33 smithi038 ceph-mon[28043]: pgmap v421: 129 pgs: 129 active+clean; 1.0 GiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 58 op/s 2024-04-22T01:14:35.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:35 smithi203 ceph-mon[31825]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 154 MiB/s wr, 50 op/s 2024-04-22T01:14:35.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:35 smithi111 ceph-mon[31057]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 154 MiB/s wr, 50 op/s 2024-04-22T01:14:35.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:35 smithi038 ceph-mon[28043]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 154 MiB/s wr, 50 op/s 2024-04-22T01:14:37.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:37 smithi111 ceph-mon[31057]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 135 MiB/s wr, 46 op/s 2024-04-22T01:14:37.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:37 smithi038 ceph-mon[28043]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 135 MiB/s wr, 46 op/s 2024-04-22T01:14:38.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:37 smithi203 ceph-mon[31825]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 135 MiB/s wr, 46 op/s 2024-04-22T01:14:38.558 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:38.558 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:38.558 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:38.558 INFO:tasks.workunit.client.0.smithi038.stdout:iozone test complete. 2024-04-22T01:14:38.559 INFO:tasks.workunit.client.0.smithi038.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-22T01:14:38.626 INFO:tasks.workunit.client.0.smithi038.stdout: Iozone: Performance Test of File I/O 2024-04-22T01:14:38.626 INFO:tasks.workunit.client.0.smithi038.stdout: Version $Revision: 3.494 $ 2024-04-22T01:14:38.626 INFO:tasks.workunit.client.0.smithi038.stdout: Compiled for 64 bit mode. 2024-04-22T01:14:38.626 INFO:tasks.workunit.client.0.smithi038.stdout: Build: linux-AMD64 2024-04-22T01:14:38.626 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:38.626 INFO:tasks.workunit.client.0.smithi038.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Run began: Mon Apr 22 01:14:38 2024 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:38.627 INFO:tasks.workunit.client.0.smithi038.stdout: Include close in write timing 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Include fsync in write timing 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: File size set to 1048576 kB 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Record Size 1024 kB 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Output is in kBytes/sec 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Time Resolution = 0.000001 seconds. 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Processor cache size set to 1024 kBytes. 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Processor cache line size set to 32 bytes. 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: File stride size set to 17 * record size. 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Throughput test with 1 process 2024-04-22T01:14:38.628 INFO:tasks.workunit.client.0.smithi038.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-04-22T01:14:39.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:39 smithi111 ceph-mon[31057]: pgmap v424: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 99 MiB/s wr, 40 op/s 2024-04-22T01:14:39.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:39 smithi038 ceph-mon[28043]: pgmap v424: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 99 MiB/s wr, 40 op/s 2024-04-22T01:14:40.009 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:40.009 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 initial writers = 777678.62 kB/sec 2024-04-22T01:14:40.010 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 initial writers = 777576.66 kB/sec 2024-04-22T01:14:40.010 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 777678.62 kB/sec 2024-04-22T01:14:40.010 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 777678.62 kB/sec 2024-04-22T01:14:40.010 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 777678.62 kB/sec 2024-04-22T01:14:40.010 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:40.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:39 smithi203 ceph-mon[31825]: pgmap v424: 129 pgs: 129 active+clean; 1.0 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 99 MiB/s wr, 40 op/s 2024-04-22T01:14:41.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:41 smithi111 ceph-mon[31057]: pgmap v425: 129 pgs: 129 active+clean; 1.1 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 60 MiB/s wr, 27 op/s 2024-04-22T01:14:41.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:41 smithi038 ceph-mon[28043]: pgmap v425: 129 pgs: 129 active+clean; 1.1 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 60 MiB/s wr, 27 op/s 2024-04-22T01:14:42.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:41 smithi203 ceph-mon[31825]: pgmap v425: 129 pgs: 129 active+clean; 1.1 GiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 60 MiB/s wr, 27 op/s 2024-04-22T01:14:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.167+0000 7f55003de640 1 -- 172.21.15.38:0/2587144671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f8103380 msgr2=0x7f54f8109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.167+0000 7f55003de640 1 --2- 172.21.15.38:0/2587144671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f8103380 0x7f54f8109c10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009ca0 tx=0x7f54f002c0f0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 -- 172.21.15.38:0/2587144671 shutdown_connections 2024-04-22T01:14:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 --2- 172.21.15.38:0/2587144671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f8103380 0x7f54f8109c10 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 --2- 172.21.15.38:0/2587144671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54f81029c0 0x7f54f8102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 --2- 172.21.15.38:0/2587144671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f81017d0 0x7f54f8101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 -- 172.21.15.38:0/2587144671 >> 172.21.15.38:0/2587144671 conn(0x7f54f80fcd60 msgr2=0x7f54f80ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 -- 172.21.15.38:0/2587144671 shutdown_connections 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 -- 172.21.15.38:0/2587144671 wait complete. 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 Processor -- start 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.168+0000 7f55003de640 1 -- start start 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54f81017d0 0x7f54f8199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 0x7f54f819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f8103380 0x7f54f81a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f54f810d450 con 0x7f54f81029c0 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f54f810d2d0 con 0x7f54f81017d0 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54f810d5d0 con 0x7f54f8103380 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 0x7f54f819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 0x7f54f819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51288/0 (socket says 172.21.15.38:51288) 2024-04-22T01:14:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 -- 172.21.15.38:0/2400121635 learned_addr learned my addr 172.21.15.38:0/2400121635 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fe17a640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54f81017d0 0x7f54f8199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fe97b640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f8103380 0x7f54f81a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 -- 172.21.15.38:0/2400121635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f8103380 msgr2=0x7f54f81a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f8103380 0x7f54f81a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 -- 172.21.15.38:0/2400121635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54f81017d0 msgr2=0x7f54f8199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54f81017d0 0x7f54f8199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 -- 172.21.15.38:0/2400121635 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f81a17e0 con 0x7f54f81029c0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54fd979640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 0x7f54f819a100 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f54e800d950 tx=0x7f54e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54ef7fe640 1 -- 172.21.15.38:0/2400121635 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e8017070 con 0x7f54f81029c0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54ef7fe640 1 -- 172.21.15.38:0/2400121635 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54e8009c30 con 0x7f54f81029c0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f54ef7fe640 1 -- 172.21.15.38:0/2400121635 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e800f6b0 con 0x7f54f81029c0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.169+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f81a1ad0 con 0x7f54f81029c0 2024-04-22T01:14:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.170+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f81a2070 con 0x7f54f81029c0 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.172+0000 7f54ef7fe640 1 -- 172.21.15.38:0/2400121635 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f54e80109d0 con 0x7f54f81029c0 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.172+0000 7f5502fca740 1 Processor -- start 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 -- start start 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8ac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54ef7fe640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f54e40a2cb0 0x7f54e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56501b810b70 con 0x56501b8abd30 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56501b7b9470 con 0x56501b8ac720 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f5502fca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56501b80cb30 con 0x56501b8b1890 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8ac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe17a640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f54e40a2cb0 0x7f54e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8ac150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51298/0 (socket says 172.21.15.38:51298) 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 -- 172.21.15.38:0/2576163095 learned_addr learned my addr 172.21.15.38:0/2576163095 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54ef7fe640 1 -- 172.21.15.38:0/2400121635 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f54e80ceef0 con 0x7f54f81029c0 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fd979640 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 -- 172.21.15.38:0/2576163095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 msgr2=0x56501b8b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 -- 172.21.15.38:0/2576163095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 msgr2=0x56501b8b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.173+0000 7f54fe97b640 1 -- 172.21.15.38:0/2576163095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56501b525210 con 0x56501b8abd30 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.174+0000 7f54fe17a640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f54e40a2cb0 0x7f54e40a5170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f54f4005fd0 tx=0x7f54f400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.174+0000 7f54fe97b640 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8ac150 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009430 tx=0x7f54f0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a4c4ddf4624d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.174+0000 7f54ecbfe640 1 -- 172.21.15.38:0/2576163095 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0048070 con 0x56501b8abd30 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.174+0000 7f54ecbfe640 1 -- 172.21.15.38:0/2576163095 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54f0003970 con 0x56501b8abd30 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.174+0000 7f54ecbfe640 1 -- 172.21.15.38:0/2576163095 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0039040 con 0x56501b8abd30 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- 172.21.15.38:0/2576163095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 msgr2=0x56501b8ac150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8ac150 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009430 tx=0x7f54f0009360 comp rx=0 tx=0).stop 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- 172.21.15.38:0/2576163095 shutdown_connections 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- 172.21.15.38:0/2576163095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8ac150 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- 172.21.15.38:0/2576163095 >> 172.21.15.38:0/2576163095 conn(0x56501b8a72a0 msgr2=0x56501b8a76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- 172.21.15.38:0/2576163095 shutdown_connections 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- 172.21.15.38:0/2576163095 wait complete. 2024-04-22T01:14:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- start start 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 Processor -- start 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8bd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56501b8b47e0 con 0x56501b8abd30 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.175+0000 7f5502fca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56501b7cca10 con 0x56501b8ac720 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f5502fca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56501b8b4960 con 0x56501b8b1890 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f5502fca740 10 client.0 _ll_register_callbacks cb 0x56501b807b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8b6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51310/0 (socket says 172.21.15.38:51310) 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 -- 172.21.15.38:0/290322046 learned_addr learned my addr 172.21.15.38:0/290322046 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fd979640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe97b640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8bd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 -- 172.21.15.38:0/290322046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 msgr2=0x56501b8bd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8bd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 -- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 msgr2=0x56501b8b6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 -- 172.21.15.38:0/290322046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56501b5f1620 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54fe17a640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8b6000 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f54f40116c0 tx=0x7f54f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cc6cb5b4bba64b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54e1ffb640 1 -- 172.21.15.38:0/290322046 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f401e830 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f54e1ffb640 1 -- 172.21.15.38:0/290322046 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f54f401e9d0 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.176+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54f0003970 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f54e1ffb640 1 -- 172.21.15.38:0/290322046 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54f401ec10 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 10 client.16089 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56501b8c09c0 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f54e1ffb640 1 -- 172.21.15.38:0/290322046 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54f0003970 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f54e1ffb640 1 -- 172.21.15.38:0/290322046 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54f401c4d0 con 0x56501b8abd30 2024-04-22T01:14:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 10 client.16089 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 20 client.16089 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8c2090 0x56501b8c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 4 client.16089 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56501b81e0b0 con 0x56501b8c2090 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.177+0000 7f54fd979640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8c2090 0x56501b8c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.178+0000 7f54e1ffb640 10 client.16089 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.178+0000 7f54fd979640 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8c2090 0x56501b8c44e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=428e76b223ecb052 server_cookie=2fa4d937a0cee4b6 in_seq=0 out_seq=0 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.178+0000 7f54e1ffb640 1 -- 172.21.15.38:0/290322046 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56501b81e0b0 con 0x56501b8c2090 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.178+0000 7f54e1ffb640 10 client.16089 handle_command_reply: tid=0 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.178+0000 7f5502fca740 1 client.16089 shutdown 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 msgr2=0x56501b8b6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8b6000 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f54f40116c0 tx=0x7f54f4004730 comp rx=0 tx=0).stop 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 shutdown_connections 2024-04-22T01:14:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8c2090 0x56501b8c44e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8b1890 0x56501b8bd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8ac720 0x56501b8b6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 --2- 172.21.15.38:0/290322046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8abd30 0x56501b8b6000 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f54e1ffb640 0 client.16089 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 >> 172.21.15.38:0/290322046 conn(0x56501b8a72a0 msgr2=0x56501b8a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 shutdown_connections 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 1 -- 172.21.15.38:0/290322046 wait complete. 2024-04-22T01:14:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.179+0000 7f5502fca740 20 client.16089 trim_cache size 0 max 16384 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 Processor -- start 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 -- start start 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81ac680 con 0x56501b8ac720 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81ae1c0 con 0x56501b8a72a0 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f5502fca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81acd50 con 0x56501b8abd30 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8a5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51316/0 (socket says 172.21.15.38:51316) 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 -- 172.21.15.38:0/1281920215 learned_addr learned my addr 172.21.15.38:0/1281920215 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fd979640 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe17a640 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 -- 172.21.15.38:0/1281920215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 msgr2=0x56501b8bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 -- 172.21.15.38:0/1281920215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 msgr2=0x56501b8b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8b64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.194+0000 7f54fe97b640 1 -- 172.21.15.38:0/1281920215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56501b8b4a60 con 0x56501b8ac720 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f54fd979640 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f54fe97b640 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8a5c30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f54f002c630 tx=0x7f54f0009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d9963e5ec9cdcb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f54e3fff640 1 -- 172.21.15.38:0/1281920215 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0048070 con 0x56501b8ac720 2024-04-22T01:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f54e3fff640 1 -- 172.21.15.38:0/1281920215 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54f0003e50 con 0x56501b8ac720 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f54e3fff640 1 -- 172.21.15.38:0/1281920215 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0039040 con 0x56501b8ac720 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 -- 172.21.15.38:0/1281920215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 msgr2=0x56501b8a5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8a5c30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f54f002c630 tx=0x7f54f0009a20 comp rx=0 tx=0).stop 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 -- 172.21.15.38:0/1281920215 shutdown_connections 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8a5c30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 --2- 172.21.15.38:0/1281920215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.195+0000 7f5502fca740 1 -- 172.21.15.38:0/1281920215 >> 172.21.15.38:0/1281920215 conn(0x56501b8ad320 msgr2=0x56501b8a7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 -- 172.21.15.38:0/1281920215 shutdown_connections 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 -- 172.21.15.38:0/1281920215 wait complete. 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 -- start start 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 Processor -- start 2024-04-22T01:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8baeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8bc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81adaf0 con 0x56501b8ac720 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81ad420 con 0x56501b8a72a0 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81aa640 con 0x56501b8abd30 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f5502fca740 10 client.0 _ll_register_callbacks cb 0x7f54f4005c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f54fe97b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8bc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f54fe97b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8bc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51324/0 (socket says 172.21.15.38:51324) 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f54fe97b640 1 -- 172.21.15.38:0/3279954955 learned_addr learned my addr 172.21.15.38:0/3279954955 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f54fd979640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.196+0000 7f54fe17a640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8baeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe97b640 1 -- 172.21.15.38:0/3279954955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 msgr2=0x56501b8bb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe97b640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe97b640 1 -- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 msgr2=0x56501b8baeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe97b640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8baeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe97b640 1 -- 172.21.15.38:0/3279954955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56501b8bc9a0 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe17a640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8baeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54fe97b640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8bc3b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009bf0 tx=0x7f54f0032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1041f155f18f1730 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54e1ffb640 1 -- 172.21.15.38:0/3279954955 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0059060 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54f0003e50 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54e1ffb640 1 -- 172.21.15.38:0/3279954955 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f54f0004400 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54e1ffb640 1 -- 172.21.15.38:0/3279954955 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54f0060050 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f5502fca740 10 client.16095 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56501b928720 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.197+0000 7f54e1ffb640 1 -- 172.21.15.38:0/3279954955 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54f0003e50 con 0x56501b8ac720 2024-04-22T01:14:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f54e1ffb640 1 -- 172.21.15.38:0/3279954955 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54f003c070 con 0x56501b8ac720 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f5502fca740 10 client.16095 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f5502fca740 10 client.16095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f5502fca740 20 client.16095 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f5502fca740 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8b1890 0x56501b92abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f5502fca740 4 client.16095 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56501b818400 con 0x56501b8b1890 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f54fe17a640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8b1890 0x56501b92abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f54fe17a640 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8b1890 0x56501b92abe0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e99404d24344de2 server_cookie=401197d185c75f60 in_seq=0 out_seq=0 2024-04-22T01:14:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.198+0000 7f54e1ffb640 10 client.16095 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.210+0000 7f54e1ffb640 1 -- 172.21.15.38:0/3279954955 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56501b818400 con 0x56501b8b1890 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.210+0000 7f54e1ffb640 10 client.16095 handle_command_reply: tid=0 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.210+0000 7f5502fca740 1 client.16095 shutdown 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 msgr2=0x56501b8bc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8bc3b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009bf0 tx=0x7f54f0032590 comp rx=0 tx=0).stop 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 shutdown_connections 2024-04-22T01:14:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56501b8b1890 0x56501b92abe0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56501b8ac720 0x56501b8bc3b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56501b8abd30 0x56501b8bb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 --2- 172.21.15.38:0/3279954955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56501b8a72a0 0x56501b8baeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f54e1ffb640 0 client.16095 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 >> 172.21.15.38:0/3279954955 conn(0x56501b8ad320 msgr2=0x56501b8aa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 shutdown_connections 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 1 -- 172.21.15.38:0/3279954955 wait complete. 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f5502fca740 20 client.16095 trim_cache size 0 max 16384 2024-04-22T01:14:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f54e40a2cb0 msgr2=0x7f54e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f54e40a2cb0 0x7f54e40a5170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f54f4005fd0 tx=0x7f54f400b390 comp rx=0 tx=0).stop 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 msgr2=0x7f54f819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 0x7f54f819a100 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f54e800d950 tx=0x7f54e800de10 comp rx=0 tx=0).stop 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 shutdown_connections 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f54e40a2cb0 0x7f54e40a5170 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f8103380 0x7f54f81a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54f81029c0 0x7f54f819a100 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.211+0000 7f55003de640 1 --2- 172.21.15.38:0/2400121635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54f81017d0 0x7f54f8199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.212+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 >> 172.21.15.38:0/2400121635 conn(0x7f54f80fcd60 msgr2=0x7f54f8073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.212+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 shutdown_connections 2024-04-22T01:14:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:42.212+0000 7f55003de640 1 -- 172.21.15.38:0/2400121635 wait complete. 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 rewriters = 870202.75 kB/sec 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 rewriters = 869707.71 kB/sec 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 870202.75 kB/sec 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 870202.75 kB/sec 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 870202.75 kB/sec 2024-04-22T01:14:43.524 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:43.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:43 smithi111 ceph-mon[31057]: pgmap v426: 129 pgs: 129 active+clean; 1.6 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 87 MiB/s wr, 38 op/s 2024-04-22T01:14:43.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:43 smithi038 ceph-mon[28043]: pgmap v426: 129 pgs: 129 active+clean; 1.6 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 87 MiB/s wr, 38 op/s 2024-04-22T01:14:44.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:43 smithi203 ceph-mon[31825]: pgmap v426: 129 pgs: 129 active+clean; 1.6 GiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 87 MiB/s wr, 38 op/s 2024-04-22T01:14:45.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:45 smithi111 ceph-mon[31057]: pgmap v427: 129 pgs: 129 active+clean; 1.8 GiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 73 MiB/s wr, 32 op/s 2024-04-22T01:14:45.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:45 smithi038 ceph-mon[28043]: pgmap v427: 129 pgs: 129 active+clean; 1.8 GiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 73 MiB/s wr, 32 op/s 2024-04-22T01:14:45.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:14:45.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:14:46.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:45 smithi203 ceph-mon[31825]: pgmap v427: 129 pgs: 129 active+clean; 1.8 GiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 73 MiB/s wr, 32 op/s 2024-04-22T01:14:46.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 -- 172.21.15.38:0/2283404661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 msgr2=0x7eff5c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 --2- 172.21.15.38:0/2283404661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c102e10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7eff44009900 tx=0x7eff4402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:46.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 -- 172.21.15.38:0/2283404661 shutdown_connections 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 --2- 172.21.15.38:0/2283404661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff5c103350 0x7eff5c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 --2- 172.21.15.38:0/2283404661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c102e10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 --2- 172.21.15.38:0/2283404661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff5c1017a0 0x7eff5c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 -- 172.21.15.38:0/2283404661 >> 172.21.15.38:0/2283404661 conn(0x7eff5c0fcd10 msgr2=0x7eff5c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 -- 172.21.15.38:0/2283404661 shutdown_connections 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.054+0000 7eff612c6640 1 -- 172.21.15.38:0/2283404661 wait complete. 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 Processor -- start 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 -- start start 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff5c1017a0 0x7eff5c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff5c103350 0x7eff5c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c10d400 con 0x7eff5c102990 2024-04-22T01:14:46.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c10d280 con 0x7eff5c1017a0 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff612c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c10d580 con 0x7eff5c103350 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51354/0 (socket says 172.21.15.38:51354) 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 -- 172.21.15.38:0/2239640103 learned_addr learned my addr 172.21.15.38:0/2239640103 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff5c103350 0x7eff5c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff5affd640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff5c1017a0 0x7eff5c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 -- 172.21.15.38:0/2239640103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff5c103350 msgr2=0x7eff5c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff5c103350 0x7eff5c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 -- 172.21.15.38:0/2239640103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff5c1017a0 msgr2=0x7eff5c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff5c1017a0 0x7eff5c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.055+0000 7eff53fff640 1 -- 172.21.15.38:0/2239640103 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5c19f6d0 con 0x7eff5c102990 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.056+0000 7eff53fff640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c197ff0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7eff44009230 tx=0x7eff44005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.056+0000 7eff58ff9640 1 -- 172.21.15.38:0/2239640103 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44041070 con 0x7eff5c102990 2024-04-22T01:14:46.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.056+0000 7eff58ff9640 1 -- 172.21.15.38:0/2239640103 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff440054e0 con 0x7eff5c102990 2024-04-22T01:14:46.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.056+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5c19f960 con 0x7eff5c102990 2024-04-22T01:14:46.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.056+0000 7eff58ff9640 1 -- 172.21.15.38:0/2239640103 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44034440 con 0x7eff5c102990 2024-04-22T01:14:46.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.056+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff5c19fea0 con 0x7eff5c102990 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.058+0000 7eff58ff9640 1 -- 172.21.15.38:0/2239640103 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7eff44044070 con 0x7eff5c102990 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.058+0000 7eff63f94740 1 Processor -- start 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 -- start start 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a43370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a48440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a48ab0 0x5613e8a4af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5613e89a8560 con 0x5613e8a43940 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5613e8950730 con 0x5613e8a48ab0 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff63f94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5613e89a4520 con 0x5613e8a42f50 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a48440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff58ff9640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff480a2c10 0x7eff480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a48440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51362/0 (socket says 172.21.15.38:51362) 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 -- 172.21.15.38:0/2247057837 learned_addr learned my addr 172.21.15.38:0/2247057837 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff53fff640 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a48ab0 0x5613e8a4af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a43370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff480a2c10 0x7eff480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 -- 172.21.15.38:0/2247057837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 msgr2=0x5613e8a43370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff58ff9640 1 -- 172.21.15.38:0/2239640103 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7eff440f1130 con 0x7eff5c102990 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a43370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 -- 172.21.15.38:0/2247057837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a48ab0 msgr2=0x5613e8a4af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a48ab0 0x5613e8a4af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5affd640 1 -- 172.21.15.38:0/2247057837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613e87867d0 con 0x5613e8a43940 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.059+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a43370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.060+0000 7eff5affd640 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a48440 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009b70 tx=0x7eff4c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbf736a51679eec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.060+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff480a2c10 0x7eff480a50d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7eff54005c40 tx=0x7eff54007220 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.060+0000 7eff517fa640 1 -- 172.21.15.38:0/2247057837 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c016070 con 0x5613e8a43940 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.060+0000 7eff517fa640 1 -- 172.21.15.38:0/2247057837 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff4c005500 con 0x5613e8a43940 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.060+0000 7eff517fa640 1 -- 172.21.15.38:0/2247057837 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c011470 con 0x5613e8a43940 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- 172.21.15.38:0/2247057837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 msgr2=0x5613e8a48440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a48440 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009b70 tx=0x7eff4c00e6c0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- 172.21.15.38:0/2247057837 shutdown_connections 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a48ab0 0x5613e8a4af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a48440 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- 172.21.15.38:0/2247057837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a43370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- 172.21.15.38:0/2247057837 >> 172.21.15.38:0/2247057837 conn(0x5613e8a3e4c0 msgr2=0x5613e8a3e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- 172.21.15.38:0/2247057837 shutdown_connections 2024-04-22T01:14:46.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- 172.21.15.38:0/2247057837 wait complete. 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- start start 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 Processor -- start 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 0x5613e8a4d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a4d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8a54a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5613e8a4ba60 con 0x5613e8a42f50 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5613e8963e00 con 0x5613e8a43940 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.061+0000 7eff63f94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5613e8a4bbe0 con 0x5613e8a48ab0 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff63f94740 10 client.0 _ll_register_callbacks cb 0x5613e899eeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 0x5613e8a4d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 0x5613e8a4d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51366/0 (socket says 172.21.15.38:51366) 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 -- 172.21.15.38:0/565679065 learned_addr learned my addr 172.21.15.38:0/565679065 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff53fff640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a4d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8a54a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 -- 172.21.15.38:0/565679065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 msgr2=0x5613e8a54a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8a54a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 -- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 msgr2=0x5613e8a4d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a4d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 -- 172.21.15.38:0/565679065 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613e86bc3f0 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8a54a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff5affd640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 0x5613e8a4d4a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7eff4c005f70 tx=0x7eff4c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4517eb09b6311ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff427fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff427fc640 1 -- 172.21.15.38:0/565679065 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c028060 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff427fc640 1 -- 172.21.15.38:0/565679065 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7eff4c005000 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.062+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff4c005500 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff427fc640 1 -- 172.21.15.38:0/565679065 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7eff4c016040 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 10 client.16107 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5613e867c0f0 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff427fc640 1 -- 172.21.15.38:0/565679065 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff4c005500 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff427fc640 1 -- 172.21.15.38:0/565679065 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7eff4c00f040 con 0x5613e8a42f50 2024-04-22T01:14:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 10 client.16107 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 10 client.16107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 20 client.16107 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8a58f30 0x5613e8a5b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 4 client.16107 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613e89b5aa0 con 0x5613e8a58f30 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.063+0000 7eff53fff640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8a58f30 0x5613e8a5b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.064+0000 7eff427fc640 10 client.16107 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.064+0000 7eff53fff640 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8a58f30 0x5613e8a5b320 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1f39e0aba4e65f8 server_cookie=3486c17023f8a6 in_seq=0 out_seq=0 2024-04-22T01:14:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.064+0000 7eff427fc640 1 -- 172.21.15.38:0/565679065 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613e89b5aa0 con 0x5613e8a58f30 2024-04-22T01:14:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.064+0000 7eff427fc640 10 client.16107 handle_command_reply: tid=0 2024-04-22T01:14:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 client.16107 shutdown 2024-04-22T01:14:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 msgr2=0x5613e8a4d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 0x5613e8a4d4a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7eff4c005f70 tx=0x7eff4c009880 comp rx=0 tx=0).stop 2024-04-22T01:14:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 shutdown_connections 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8a58f30 0x5613e8a5b320 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8a54a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a4d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff427fc640 0 client.16107 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 --2- 172.21.15.38:0/565679065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a42f50 0x5613e8a4d4a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 >> 172.21.15.38:0/565679065 conn(0x5613e8a3e4c0 msgr2=0x5613e8a3d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 shutdown_connections 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 1 -- 172.21.15.38:0/565679065 wait complete. 2024-04-22T01:14:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.065+0000 7eff63f94740 20 client.16107 trim_cache size 0 max 16384 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 Processor -- start 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 -- start start 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a3e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a549e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 0x5613e8a5fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1aa530 con 0x5613e8a48ab0 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1ac070 con 0x5613e8a43940 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff63f94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1aac00 con 0x5613e8a42f50 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 0x5613e8a5fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 0x5613e8a5fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51368/0 (socket says 172.21.15.38:51368) 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 -- 172.21.15.38:0/1791495961 learned_addr learned my addr 172.21.15.38:0/1791495961 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff53fff640 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 -- 172.21.15.38:0/1791495961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 msgr2=0x5613e8a3e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a3e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 -- 172.21.15.38:0/1791495961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 msgr2=0x5613e8a549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 -- 172.21.15.38:0/1791495961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613e8a4bd60 con 0x5613e8a48ab0 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.083+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 0x5613e8a5fb90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7eff540117e0 tx=0x7eff5400b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d4fb34a01ec13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff50ff9640 1 -- 172.21.15.38:0/1791495961 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5401d990 con 0x5613e8a48ab0 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff50ff9640 1 -- 172.21.15.38:0/1791495961 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff54003e50 con 0x5613e8a48ab0 2024-04-22T01:14:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff50ff9640 1 -- 172.21.15.38:0/1791495961 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54004630 con 0x5613e8a48ab0 2024-04-22T01:14:46.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 -- 172.21.15.38:0/1791495961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 msgr2=0x5613e8a5fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 0x5613e8a5fb90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7eff540117e0 tx=0x7eff5400b2e0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 -- 172.21.15.38:0/1791495961 shutdown_connections 2024-04-22T01:14:46.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a48ab0 0x5613e8a5fb90 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a43940 0x5613e8a549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 --2- 172.21.15.38:0/1791495961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a42f50 0x5613e8a3e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.084+0000 7eff63f94740 1 -- 172.21.15.38:0/1791495961 >> 172.21.15.38:0/1791495961 conn(0x5613e8a3e4c0 msgr2=0x5613e8a41fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 -- 172.21.15.38:0/1791495961 shutdown_connections 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 -- 172.21.15.38:0/1791495961 wait complete. 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 -- start start 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 Processor -- start 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a42f50 0x5613e8a51610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a51b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8abd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1ab9a0 con 0x5613e8a43940 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1ab2d0 con 0x5613e8a42f50 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1a84f0 con 0x5613e8a48ab0 2024-04-22T01:14:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff63f94740 10 client.0 _ll_register_callbacks cb 0x7eff4c0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a51b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a51b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51376/0 (socket says 172.21.15.38:51376) 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 -- 172.21.15.38:0/3244519530 learned_addr learned my addr 172.21.15.38:0/3244519530 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff5affd640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a42f50 0x5613e8a51610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff5b7fe640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8abd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 -- 172.21.15.38:0/3244519530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 msgr2=0x5613e8abd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8abd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 -- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a42f50 msgr2=0x5613e8a51610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.085+0000 7eff53fff640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a42f50 0x5613e8a51610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff53fff640 1 -- 172.21.15.38:0/3244519530 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613e8a54bd0 con 0x5613e8a43940 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff53fff640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a51b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7eff44009430 tx=0x7eff4408d470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccd9a1089455977 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff427fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff427fc640 1 -- 172.21.15.38:0/3244519530 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4404b040 con 0x5613e8a43940 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff427fc640 1 -- 172.21.15.38:0/3244519530 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7eff440bd820 con 0x5613e8a43940 2024-04-22T01:14:46.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff54003e50 con 0x5613e8a43940 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff427fc640 1 -- 172.21.15.38:0/3244519530 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7eff44004c60 con 0x5613e8a43940 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 10 client.16113 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5613e8abe950 con 0x5613e8a43940 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff427fc640 1 -- 172.21.15.38:0/3244519530 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff54003e50 con 0x5613e8a43940 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff427fc640 1 -- 172.21.15.38:0/3244519530 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7eff4c009c60 con 0x5613e8a43940 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 10 client.16113 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 10 client.16113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 20 client.16113 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8abf330 0x5613e8ac1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 4 client.16113 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.086+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5613e89afdf0 con 0x5613e8abf330 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff5affd640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8abf330 0x5613e8ac1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff5affd640 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8abf330 0x5613e8ac1720 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f22fd0fcf9c7aab1 server_cookie=5b8a6c734b83d862 in_seq=0 out_seq=0 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff427fc640 10 client.16113 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff427fc640 1 -- 172.21.15.38:0/3244519530 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5613e89afdf0 con 0x5613e8abf330 2024-04-22T01:14:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff427fc640 10 client.16113 handle_command_reply: tid=0 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 client.16113 shutdown 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 msgr2=0x5613e8a51b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a51b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7eff44009430 tx=0x7eff4408d470 comp rx=0 tx=0).stop 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 shutdown_connections 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5613e8abf330 0x5613e8ac1720 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613e8a48ab0 0x5613e8abd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613e8a43940 0x5613e8a51b50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff427fc640 0 client.16113 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 --2- 172.21.15.38:0/3244519530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5613e8a42f50 0x5613e8a51610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 >> 172.21.15.38:0/3244519530 conn(0x5613e8a3e4c0 msgr2=0x5613e8a41fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 shutdown_connections 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.087+0000 7eff63f94740 1 -- 172.21.15.38:0/3244519530 wait complete. 2024-04-22T01:14:46.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff63f94740 20 client.16113 trim_cache size 0 max 16384 2024-04-22T01:14:46.094 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:14:46.094 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T01:14:46.094 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T01:14:46.094 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff480a2c10 msgr2=0x7eff480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff480a2c10 0x7eff480a50d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7eff54005c40 tx=0x7eff54007220 comp rx=0 tx=0).stop 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 msgr2=0x7eff5c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c197ff0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7eff44009230 tx=0x7eff44005dd0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 shutdown_connections 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7eff480a2c10 0x7eff480a50d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff5c103350 0x7eff5c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff5c102990 0x7eff5c197ff0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 --2- 172.21.15.38:0/2239640103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff5c1017a0 0x7eff5c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 >> 172.21.15.38:0/2239640103 conn(0x7eff5c0fcd10 msgr2=0x7eff5c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 shutdown_connections 2024-04-22T01:14:46.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.088+0000 7eff612c6640 1 -- 172.21.15.38:0/2239640103 wait complete. 2024-04-22T01:14:46.100 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T01:14:46.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:14:46.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 -- 172.21.15.38:0/3021690224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 msgr2=0x7f6084104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 --2- 172.21.15.38:0/3021690224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084104670 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6078009900 tx=0x7f607802bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 -- 172.21.15.38:0/3021690224 shutdown_connections 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 --2- 172.21.15.38:0/3021690224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 0x7f60841098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 --2- 172.21.15.38:0/3021690224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6084104bb0 0x7f6084106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 --2- 172.21.15.38:0/3021690224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084104670 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 -- 172.21.15.38:0/3021690224 >> 172.21.15.38:0/3021690224 conn(0x7f60840fbc30 msgr2=0x7f60840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 -- 172.21.15.38:0/3021690224 shutdown_connections 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.202+0000 7f608ab8c640 1 -- 172.21.15.38:0/3021690224 wait complete. 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 Processor -- start 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 -- start start 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6084104bb0 0x7f6084100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 0x7f6084100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f608410d410 con 0x7f6084102280 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f608410d290 con 0x7f60841074e0 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f608ab8c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f608410d590 con 0x7f6084104bb0 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51394/0 (socket says 172.21.15.38:51394) 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 -- 172.21.15.38:0/551217905 learned_addr learned my addr 172.21.15.38:0/551217905 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6083fff640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6084104bb0 0x7f6084100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6089129640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 0x7f6084100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 -- 172.21.15.38:0/551217905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6084104bb0 msgr2=0x7f6084100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6084104bb0 0x7f6084100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 -- 172.21.15.38:0/551217905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 msgr2=0x7f6084100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 0x7f6084100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6088928640 1 -- 172.21.15.38:0/551217905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6084101000 con 0x7f6084102280 2024-04-22T01:14:46.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.203+0000 7f6089129640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 0x7f6084100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:46.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.204+0000 7f6088928640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084101bf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6078009230 tx=0x7f6078005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.204+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078041070 con 0x7f6084102280 2024-04-22T01:14:46.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.204+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60841a1b10 con 0x7f6084102280 2024-04-22T01:14:46.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.204+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60780054e0 con 0x7f6084102280 2024-04-22T01:14:46.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.204+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078034440 con 0x7f6084102280 2024-04-22T01:14:46.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.204+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60841a20b0 con 0x7f6084102280 2024-04-22T01:14:46.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.205+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60840fd720 con 0x7f6084102280 2024-04-22T01:14:46.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.206+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6078044070 con 0x7f6084102280 2024-04-22T01:14:46.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.207+0000 7f6081ffb640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60740a2b90 0x7f60740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.207+0000 7f6083fff640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60740a2b90 0x7f60740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.207+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f60780f1630 con 0x7f6084102280 2024-04-22T01:14:46.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.207+0000 7f6083fff640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60740a2b90 0x7f60740a5050 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6070005d00 tx=0x7f6070005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.210+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f60780be180 con 0x7f6084102280 2024-04-22T01:14:46.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.433+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f608402d170 con 0x7f6084102280 2024-04-22T01:14:46.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.434+0000 7f6081ffb640 1 -- 172.21.15.38:0/551217905 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f60780bda80 con 0x7f6084102280 2024-04-22T01:14:46.434 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:14:46.434 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:14:46.434 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60740a2b90 msgr2=0x7f60740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60740a2b90 0x7f60740a5050 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6070005d00 tx=0x7f6070005c50 comp rx=0 tx=0).stop 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 msgr2=0x7f6084101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084101bf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6078009230 tx=0x7f6078005dd0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 shutdown_connections 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f60740a2b90 0x7f60740a5050 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60841074e0 0x7f6084100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6084104bb0 0x7f6084100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 --2- 172.21.15.38:0/551217905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6084102280 0x7f6084101bf0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 >> 172.21.15.38:0/551217905 conn(0x7f60840fbc30 msgr2=0x7f6084102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 shutdown_connections 2024-04-22T01:14:46.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.436+0000 7f608ab8c640 1 -- 172.21.15.38:0/551217905 wait complete. 2024-04-22T01:14:46.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 -- 172.21.15.38:0/498064130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788102220 msgr2=0x7f1788104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 --2- 172.21.15.38:0/498064130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788102220 0x7f1788104610 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1778009900 tx=0x7f177802bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 -- 172.21.15.38:0/498064130 shutdown_connections 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 --2- 172.21.15.38:0/498064130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1788107480 0x7f17881098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 --2- 172.21.15.38:0/498064130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1788104b50 0x7f1788106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 --2- 172.21.15.38:0/498064130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788102220 0x7f1788104610 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 -- 172.21.15.38:0/498064130 >> 172.21.15.38:0/498064130 conn(0x7f17880fbc30 msgr2=0x7f17880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 -- 172.21.15.38:0/498064130 shutdown_connections 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.570+0000 7f178dad3640 1 -- 172.21.15.38:0/498064130 wait complete. 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 Processor -- start 2024-04-22T01:14:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 -- start start 2024-04-22T01:14:46.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1788102220 0x7f1788197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 0x7f17881980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1788107480 0x7f178819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f178810d4a0 con 0x7f1788104b50 2024-04-22T01:14:46.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f178810d320 con 0x7f1788102220 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f178dad3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f178810d620 con 0x7f1788107480 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 0x7f17881980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 0x7f17881980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51416/0 (socket says 172.21.15.38:51416) 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f1786ffd640 1 -- 172.21.15.38:0/1860549108 learned_addr learned my addr 172.21.15.38:0/1860549108 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f17877fe640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1788102220 0x7f1788197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.571+0000 7f1787fff640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1788107480 0x7f178819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1786ffd640 1 -- 172.21.15.38:0/1860549108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1788107480 msgr2=0x7f178819f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1786ffd640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1788107480 0x7f178819f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1786ffd640 1 -- 172.21.15.38:0/1860549108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1788102220 msgr2=0x7f1788197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1786ffd640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1788102220 0x7f1788197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1786ffd640 1 -- 172.21.15.38:0/1860549108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178819f7a0 con 0x7f1788104b50 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1786ffd640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 0x7f17881980c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f177c00e9f0 tx=0x7f177c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1784ff9640 1 -- 172.21.15.38:0/1860549108 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c00ccc0 con 0x7f1788104b50 2024-04-22T01:14:46.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1784ff9640 1 -- 172.21.15.38:0/1860549108 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177c005500 con 0x7f1788104b50 2024-04-22T01:14:46.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f178819fa90 con 0x7f1788104b50 2024-04-22T01:14:46.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f1784ff9640 1 -- 172.21.15.38:0/1860549108 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c010690 con 0x7f1788104b50 2024-04-22T01:14:46.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.572+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f178819ff90 con 0x7f1788104b50 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1784ff9640 1 -- 172.21.15.38:0/1860549108 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f177c0046c0 con 0x7f1788104b50 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 Processor -- start 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 -- start start 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 0x55e207431870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e207436940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207436fb0 0x55e207439440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e207392530 con 0x55e207436fb0 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e20733a830 con 0x55e207431e40 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1790795740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e20738e4f0 con 0x55e207431450 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.575+0000 7f1784ff9640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17700a2b90 0x7f17700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207436fb0 0x55e207439440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 0x55e207431870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 0x55e207431870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34058/0 (socket says 172.21.15.38:34058) 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 -- 172.21.15.38:0/2049161562 learned_addr learned my addr 172.21.15.38:0/2049161562 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17700a2b90 0x7f17700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f17877fe640 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e207436940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1784ff9640 1 -- 172.21.15.38:0/1860549108 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f177c0cf780 con 0x7f1788104b50 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 -- 172.21.15.38:0/2049161562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 msgr2=0x55e207436940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e207436940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 -- 172.21.15.38:0/2049161562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207436fb0 msgr2=0x55e207439440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207436fb0 0x55e207439440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 -- 172.21.15.38:0/2049161562 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e20735e2c0 con 0x55e207431450 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1786ffd640 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207436fb0 0x55e207439440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17700a2b90 0x7f17700a5050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1780010c40 tx=0x7f1780017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.576+0000 7f1787fff640 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 0x55e207431870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1780012d80 tx=0x7f1780010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de54d002aaf91e6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.577+0000 7f176dffb640 1 -- 172.21.15.38:0/2049161562 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780023070 con 0x55e207431450 2024-04-22T01:14:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.577+0000 7f176dffb640 1 -- 172.21.15.38:0/2049161562 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1780012e70 con 0x55e207431450 2024-04-22T01:14:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.577+0000 7f176dffb640 1 -- 172.21.15.38:0/2049161562 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780003bd0 con 0x55e207431450 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 -- 172.21.15.38:0/2049161562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 msgr2=0x55e207431870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 0x55e207431870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1780012d80 tx=0x7f1780010740 comp rx=0 tx=0).stop 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 -- 172.21.15.38:0/2049161562 shutdown_connections 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207436fb0 0x55e207439440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e207436940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 --2- 172.21.15.38:0/2049161562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207431450 0x55e207431870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 -- 172.21.15.38:0/2049161562 >> 172.21.15.38:0/2049161562 conn(0x55e2074285b0 msgr2=0x55e2074289b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 -- 172.21.15.38:0/2049161562 shutdown_connections 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 -- 172.21.15.38:0/2049161562 wait complete. 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.578+0000 7f1790795740 1 -- start start 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 Processor -- start 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431450 0x55e207434360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 0x55e2074348a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e2074358b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e20742cee0 con 0x55e207431e40 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e20734df10 con 0x55e207431450 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e20742d060 con 0x55e207436fb0 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.579+0000 7f1790795740 10 client.0 _ll_register_callbacks cb 0x55e2070a5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 0x55e2074348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 0x55e2074348a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51426/0 (socket says 172.21.15.38:51426) 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 -- 172.21.15.38:0/872816724 learned_addr learned my addr 172.21.15.38:0/872816724 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1787fff640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e2074358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f17877fe640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431450 0x55e207434360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 -- 172.21.15.38:0/872816724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 msgr2=0x55e2074358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e2074358b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 -- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431450 msgr2=0x55e207434360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431450 0x55e207434360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 -- 172.21.15.38:0/872816724 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e20731fc80 con 0x55e207431e40 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1786ffd640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 0x55e2074348a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f177c0985e0 tx=0x7f177c06dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387e5530fa20844 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1766ffd640 1 -- 172.21.15.38:0/872816724 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c09c310 con 0x55e207431e40 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1766ffd640 1 -- 172.21.15.38:0/872816724 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f177c06d960 con 0x55e207431e40 2024-04-22T01:14:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1780012e70 con 0x55e207431e40 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1766ffd640 1 -- 172.21.15.38:0/872816724 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f177c09bc10 con 0x55e207431e40 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.580+0000 7f1790795740 10 client.16131 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2070a63f0 con 0x55e207431e40 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1766ffd640 1 -- 172.21.15.38:0/872816724 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1780012e70 con 0x55e207431e40 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1766ffd640 1 -- 172.21.15.38:0/872816724 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f177c0cfbb0 con 0x55e207431e40 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 10 client.16131 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 10 client.16131 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 20 client.16131 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e207442f60 0x55e2074453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 4 client.16131 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e20739fa70 con 0x55e207442f60 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.581+0000 7f17877fe640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e207442f60 0x55e2074453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.582+0000 7f17877fe640 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e207442f60 0x55e2074453b0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=830afc8a0069efda server_cookie=4b5752ce769eef12 in_seq=0 out_seq=0 2024-04-22T01:14:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.582+0000 7f1766ffd640 10 client.16131 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1766ffd640 1 -- 172.21.15.38:0/872816724 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e20739fa70 con 0x55e207442f60 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1766ffd640 10 client.16131 handle_command_reply: tid=0 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 client.16131 shutdown 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 msgr2=0x55e2074348a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 0x55e2074348a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f177c0985e0 tx=0x7f177c06dc20 comp rx=0 tx=0).stop 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 shutdown_connections 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e207442f60 0x55e2074453b0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e2074358b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1766ffd640 0 client.16131 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431e40 0x55e2074348a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 --2- 172.21.15.38:0/872816724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431450 0x55e207434360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 >> 172.21.15.38:0/872816724 conn(0x55e2074285b0 msgr2=0x55e207427850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 shutdown_connections 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 1 -- 172.21.15.38:0/872816724 wait complete. 2024-04-22T01:14:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.583+0000 7f1790795740 20 client.16131 trim_cache size 0 max 16384 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 Processor -- start 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 -- start start 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20742d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20742d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20742de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a8320 con 0x55e207431450 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a9e60 con 0x55e207431e40 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1790795740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a89f0 con 0x55e207436fb0 2024-04-22T01:14:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f17877fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20742d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f17877fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20742d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51432/0 (socket says 172.21.15.38:51432) 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f17877fe640 1 -- 172.21.15.38:0/3052865890 learned_addr learned my addr 172.21.15.38:0/3052865890 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1786ffd640 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20742d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1787fff640 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20742de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1786ffd640 1 -- 172.21.15.38:0/3052865890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 msgr2=0x55e20742de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1786ffd640 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20742de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1786ffd640 1 -- 172.21.15.38:0/3052865890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 msgr2=0x55e20742d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1786ffd640 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20742d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.616+0000 7f1786ffd640 1 -- 172.21.15.38:0/3052865890 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2071707d0 con 0x55e207431e40 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.617+0000 7f17877fe640 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20742d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.617+0000 7f1786ffd640 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20742d940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f177c098070 tx=0x7f177c004aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e2f2ecc1e96efb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.617+0000 7f176d7fa640 1 -- 172.21.15.38:0/3052865890 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c09b750 con 0x55e207431e40 2024-04-22T01:14:46.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.617+0000 7f176d7fa640 1 -- 172.21.15.38:0/3052865890 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177c09c310 con 0x55e207431e40 2024-04-22T01:14:46.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.617+0000 7f176d7fa640 1 -- 172.21.15.38:0/3052865890 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c097eb0 con 0x55e207431e40 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 -- 172.21.15.38:0/3052865890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 msgr2=0x55e20742d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20742d940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f177c098070 tx=0x7f177c004aa0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 -- 172.21.15.38:0/3052865890 shutdown_connections 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20742de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20742d940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 --2- 172.21.15.38:0/3052865890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20742d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 -- 172.21.15.38:0/3052865890 >> 172.21.15.38:0/3052865890 conn(0x55e2074285b0 msgr2=0x55e207427850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 -- 172.21.15.38:0/3052865890 shutdown_connections 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 -- 172.21.15.38:0/3052865890 wait complete. 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 -- start start 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.618+0000 7f1790795740 1 Processor -- start 2024-04-22T01:14:46.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20743d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20743d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20743e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a9790 con 0x55e207431450 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a90c0 con 0x55e207431e40 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17881a62e0 con 0x55e207436fb0 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1790795740 10 client.0 _ll_register_callbacks cb 0x7f177c00ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20743d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20743d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51444/0 (socket says 172.21.15.38:51444) 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 -- 172.21.15.38:0/2978203973 learned_addr learned my addr 172.21.15.38:0/2978203973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1786ffd640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20743d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1787fff640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20743e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 -- 172.21.15.38:0/2978203973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 msgr2=0x55e20743e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20743e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 -- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 msgr2=0x55e20743d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20743d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 -- 172.21.15.38:0/2978203973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e20743ed70 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1787fff640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20743e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f17877fe640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20743d230 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1778005c40 tx=0x7f177802caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90644eca1152840b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.619+0000 7f1766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1766ffd640 1 -- 172.21.15.38:0/2978203973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778058070 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1766ffd640 1 -- 172.21.15.38:0/2978203973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f1778042a50 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f177c09c310 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1766ffd640 1 -- 172.21.15.38:0/2978203973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1778063050 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 10 client.16137 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2074a9860 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1766ffd640 1 -- 172.21.15.38:0/2978203973 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177c09c310 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1766ffd640 1 -- 172.21.15.38:0/2978203973 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f17780531e0 con 0x55e207431450 2024-04-22T01:14:46.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 10 client.16137 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 10 client.16137 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 20 client.16137 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e2074aa260 0x55e2074ac680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 4 client.16137 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.620+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e207399dc0 con 0x55e2074aa260 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1786ffd640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e2074aa260 0x55e2074ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1786ffd640 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e2074aa260 0x55e2074ac680 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ac8022166e7e1c23 server_cookie=24a2636dafd26f07 in_seq=0 out_seq=0 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1766ffd640 10 client.16137 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1766ffd640 1 -- 172.21.15.38:0/2978203973 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e207399dc0 con 0x55e2074aa260 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1766ffd640 10 client.16137 handle_command_reply: tid=0 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1790795740 1 client.16137 shutdown 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.621+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 msgr2=0x55e20743d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20743d230 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1778005c40 tx=0x7f177802caa0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 shutdown_connections 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55e2074aa260 0x55e2074ac680 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207436fb0 0x55e20743e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1766ffd640 0 client.16137 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e207431e40 0x55e20743d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 --2- 172.21.15.38:0/2978203973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e207431450 0x55e20743d230 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 >> 172.21.15.38:0/2978203973 conn(0x55e2074285b0 msgr2=0x55e20742b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 shutdown_connections 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 1 -- 172.21.15.38:0/2978203973 wait complete. 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f1790795740 20 client.16137 trim_cache size 0 max 16384 2024-04-22T01:14:46.627 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:46.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17700a2b90 msgr2=0x7f17700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17700a2b90 0x7f17700a5050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1780010c40 tx=0x7f1780017000 comp rx=0 tx=0).stop 2024-04-22T01:14:46.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 msgr2=0x7f17881980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 0x7f17881980c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f177c00e9f0 tx=0x7f177c00eeb0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 shutdown_connections 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f17700a2b90 0x7f17700a5050 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1788107480 0x7f178819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1788104b50 0x7f17881980c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 --2- 172.21.15.38:0/1860549108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1788102220 0x7f1788197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.622+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 >> 172.21.15.38:0/1860549108 conn(0x7f17880fbc30 msgr2=0x7f1788102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.623+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 shutdown_connections 2024-04-22T01:14:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.623+0000 7f178dad3640 1 -- 172.21.15.38:0/1860549108 wait complete. 2024-04-22T01:14:46.629 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:46.629 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 readers = 8443658.00 kB/sec 2024-04-22T01:14:46.630 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 readers = 8430693.19 kB/sec 2024-04-22T01:14:46.630 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 8443658.00 kB/sec 2024-04-22T01:14:46.630 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 8443658.00 kB/sec 2024-04-22T01:14:46.630 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 8443658.00 kB/sec 2024-04-22T01:14:46.630 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:46.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T01:14:46.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 -- 172.21.15.38:0/3576033649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 msgr2=0x7fd630100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 --2- 172.21.15.38:0/3576033649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd630100ac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd620009900 tx=0x7fd62002bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 -- 172.21.15.38:0/3576033649 shutdown_connections 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 --2- 172.21.15.38:0/3576033649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630102250 0x7fd630108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 --2- 172.21.15.38:0/3576033649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630101890 0x7fd630101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 --2- 172.21.15.38:0/3576033649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd630100ac0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 -- 172.21.15.38:0/3576033649 >> 172.21.15.38:0/3576033649 conn(0x7fd6300fbc30 msgr2=0x7fd6300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 -- 172.21.15.38:0/3576033649 shutdown_connections 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.739+0000 7fd63482e640 1 -- 172.21.15.38:0/3576033649 wait complete. 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 Processor -- start 2024-04-22T01:14:46.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 -- start start 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd63006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630101890 0x7fd63019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630102250 0x7fd63019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010c370 con 0x7fd6301006a0 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010c1f0 con 0x7fd630102250 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd63482e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010c4f0 con 0x7fd630101890 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd62e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd63006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd62e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd63006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57470/0 (socket says 172.21.15.38:57470) 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd62e59c640 1 -- 172.21.15.38:0/2574501178 learned_addr learned my addr 172.21.15.38:0/2574501178 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630101890 0x7fd63019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.740+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630102250 0x7fd63019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd62e59c640 1 -- 172.21.15.38:0/2574501178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630101890 msgr2=0x7fd63019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd62e59c640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630101890 0x7fd63019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd62e59c640 1 -- 172.21.15.38:0/2574501178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630102250 msgr2=0x7fd63019ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd62e59c640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630102250 0x7fd63019ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd62e59c640 1 -- 172.21.15.38:0/2574501178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63019f6f0 con 0x7fd6301006a0 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd62e59c640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd63006a8d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd620000c00 tx=0x7fd620005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd61f7fe640 1 -- 172.21.15.38:0/2574501178 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620043020 con 0x7fd6301006a0 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd61f7fe640 1 -- 172.21.15.38:0/2574501178 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6200054e0 con 0x7fd6301006a0 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63019f980 con 0x7fd6301006a0 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd61f7fe640 1 -- 172.21.15.38:0/2574501178 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620003e60 con 0x7fd6301006a0 2024-04-22T01:14:46.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.741+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63019ff20 con 0x7fd6301006a0 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.743+0000 7fd61f7fe640 1 -- 172.21.15.38:0/2574501178 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd620046020 con 0x7fd6301006a0 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 Processor -- start 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 -- start start 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb69240530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb69243030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bb691a0530 con 0x55bb6923ba30 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bb69148830 con 0x55bb69240ba0 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd637414740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bb6919c4f0 con 0x55bb6923b040 2024-04-22T01:14:46.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd61f7fe640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6180a2c10 0x7fd6180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd62e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb69240530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.744+0000 7fd62ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:46116/0 (socket says 172.21.15.38:46116) 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3884301068 learned_addr learned my addr 172.21.15.38:0/3884301068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6180a2c10 0x7fd6180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb69243030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd61f7fe640 1 -- 172.21.15.38:0/2574501178 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd6200f14f0 con 0x7fd6301006a0 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3884301068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 msgr2=0x55bb69243030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb69243030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3884301068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 msgr2=0x55bb69240530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb69240530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3884301068 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6916c2c0 con 0x55bb6923b040 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb69243030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6180a2c10 0x7fd6180a50d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd628010c40 tx=0x7fd62800c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923b460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd628012d00 tx=0x7fd628010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5e8c6ddb310e48d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd61cff9640 1 -- 172.21.15.38:0/3884301068 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628024070 con 0x55bb6923b040 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd61cff9640 1 -- 172.21.15.38:0/3884301068 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd628012520 con 0x55bb6923b040 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.745+0000 7fd61cff9640 1 -- 172.21.15.38:0/3884301068 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628005260 con 0x55bb6923b040 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 -- 172.21.15.38:0/3884301068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 msgr2=0x55bb6923b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923b460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd628012d00 tx=0x7fd628010740 comp rx=0 tx=0).stop 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 -- 172.21.15.38:0/3884301068 shutdown_connections 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb69243030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb69240530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 --2- 172.21.15.38:0/3884301068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923b460 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 -- 172.21.15.38:0/3884301068 >> 172.21.15.38:0/3884301068 conn(0x55bb692365b0 msgr2=0x55bb692369b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 -- 172.21.15.38:0/3884301068 shutdown_connections 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 -- 172.21.15.38:0/3884301068 wait complete. 2024-04-22T01:14:46.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.746+0000 7fd637414740 1 -- start start 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 Processor -- start 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb69245220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb69245760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 0x55bb6924c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bb69243af0 con 0x55bb69240ba0 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bb6915bf10 con 0x55bb6923ba30 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bb69243c70 con 0x55bb6923b040 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd637414740 10 client.0 _ll_register_callbacks cb 0x55bb68eb3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 0x55bb6924c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 0x55bb6924c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57502/0 (socket says 172.21.15.38:57502) 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3553165975 learned_addr learned my addr 172.21.15.38:0/3553165975 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb69245760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb69245220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 msgr2=0x55bb69245220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb69245220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3553165975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 msgr2=0x55bb69245760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb69245760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 -- 172.21.15.38:0/3553165975 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb6912dc80 con 0x55bb69240ba0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb69245220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 0x55bb6924c7a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd628009a40 tx=0x7fd628012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1866337759e161b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd615ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.747+0000 7fd615ffb640 1 -- 172.21.15.38:0/3553165975 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628024040 con 0x55bb69240ba0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd615ffb640 1 -- 172.21.15.38:0/3553165975 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd628004d10 con 0x55bb69240ba0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd628012520 con 0x55bb69240ba0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd615ffb640 1 -- 172.21.15.38:0/3553165975 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd62803a050 con 0x55bb69240ba0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 10 client.16149 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bb68eb43f0 con 0x55bb69240ba0 2024-04-22T01:14:46.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd615ffb640 1 -- 172.21.15.38:0/3553165975 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd628012520 con 0x55bb69240ba0 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd615ffb640 1 -- 172.21.15.38:0/3553165975 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd62800f040 con 0x55bb69240ba0 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 10 client.16149 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 10 client.16149 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 20 client.16149 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb69250e10 0x55bb69253200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 4 client.16149 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.748+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb691ada70 con 0x55bb69250e10 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.749+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb69250e10 0x55bb69253200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.749+0000 7fd615ffb640 10 client.16149 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.749+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb69250e10 0x55bb69253200 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e7af3859a280598 server_cookie=34807126670821c8 in_seq=0 out_seq=0 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd615ffb640 1 -- 172.21.15.38:0/3553165975 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb691ada70 con 0x55bb69250e10 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd615ffb640 10 client.16149 handle_command_reply: tid=0 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 client.16149 shutdown 2024-04-22T01:14:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 msgr2=0x55bb6924c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 0x55bb6924c7a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd628009a40 tx=0x7fd628012eb0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 shutdown_connections 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb69250e10 0x55bb69253200 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb69240ba0 0x55bb6924c7a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd615ffb640 0 client.16149 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb69245760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 --2- 172.21.15.38:0/3553165975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb69245220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 >> 172.21.15.38:0/3553165975 conn(0x55bb692365b0 msgr2=0x55bb69235850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 shutdown_connections 2024-04-22T01:14:46.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 1 -- 172.21.15.38:0/3553165975 wait complete. 2024-04-22T01:14:46.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.750+0000 7fd637414740 20 client.16149 trim_cache size 0 max 16384 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.763+0000 7fd637414740 1 Processor -- start 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.763+0000 7fd637414740 1 -- start start 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.763+0000 7fd637414740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd637414740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb6923c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd637414740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb6923ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd637414740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301aa530 con 0x55bb6923ba30 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd637414740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301ac070 con 0x55bb69240ba0 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd637414740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301aac00 con 0x55bb6923b040 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb6923c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb6923c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57516/0 (socket says 172.21.15.38:57516) 2024-04-22T01:14:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 -- 172.21.15.38:0/2186620560 learned_addr learned my addr 172.21.15.38:0/2186620560 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62ed9d640 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb6923ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 -- 172.21.15.38:0/2186620560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 msgr2=0x55bb6923c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 -- 172.21.15.38:0/2186620560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 msgr2=0x55bb6923ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb6923ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 -- 172.21.15.38:0/2186620560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb68f7e7d0 con 0x55bb6923ba30 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb6923c970 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd624005fd0 tx=0x7fd624012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ac275f67ecba21 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd617fff640 1 -- 172.21.15.38:0/2186620560 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624004020 con 0x55bb6923ba30 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd617fff640 1 -- 172.21.15.38:0/2186620560 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6240041c0 con 0x55bb6923ba30 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.764+0000 7fd617fff640 1 -- 172.21.15.38:0/2186620560 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624010720 con 0x55bb6923ba30 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- 172.21.15.38:0/2186620560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 msgr2=0x55bb6923c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb6923c970 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd624005fd0 tx=0x7fd624012820 comp rx=0 tx=0).stop 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- 172.21.15.38:0/2186620560 shutdown_connections 2024-04-22T01:14:46.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb69240ba0 0x55bb6923ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923ba30 0x55bb6923c970 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- 172.21.15.38:0/2186620560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb6923b040 0x55bb6923c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- 172.21.15.38:0/2186620560 >> 172.21.15.38:0/2186620560 conn(0x55bb692365b0 msgr2=0x55bb6923a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- 172.21.15.38:0/2186620560 shutdown_connections 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- 172.21.15.38:0/2186620560 wait complete. 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- start start 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 Processor -- start 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 0x55bb692b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb692b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb69240ba0 0x55bb692b9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301ab9a0 con 0x55bb6923b040 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301ab2d0 con 0x55bb6923ba30 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301a84f0 con 0x55bb69240ba0 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd637414740 10 client.0 _ll_register_callbacks cb 0x7fd628012b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd62e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 0x55bb692b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd62e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 0x55bb692b21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57528/0 (socket says 172.21.15.38:57528) 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.765+0000 7fd62e59c640 1 -- 172.21.15.38:0/3200326209 learned_addr learned my addr 172.21.15.38:0/3200326209 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb692b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62e59c640 1 -- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb69240ba0 msgr2=0x55bb692b9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb69240ba0 0x55bb692b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62e59c640 1 -- 172.21.15.38:0/3200326209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 msgr2=0x55bb692b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb692b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62e59c640 1 -- 172.21.15.38:0/3200326209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb692b9d30 con 0x55bb6923b040 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb692b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd62e59c640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 0x55bb692b21c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd62003c040 tx=0x7fd62008c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553fd9a850d78240 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd615ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd615ffb640 1 -- 172.21.15.38:0/3200326209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620005690 con 0x55bb6923b040 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6240041c0 con 0x55bb6923b040 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd615ffb640 1 -- 172.21.15.38:0/3200326209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd6200eed30 con 0x55bb6923b040 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd615ffb640 1 -- 172.21.15.38:0/3200326209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6200be080 con 0x55bb6923b040 2024-04-22T01:14:46.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd637414740 10 client.16155 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bb692bb100 con 0x55bb6923b040 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd615ffb640 1 -- 172.21.15.38:0/3200326209 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6240041c0 con 0x55bb6923b040 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.766+0000 7fd615ffb640 1 -- 172.21.15.38:0/3200326209 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd6200b9ff0 con 0x55bb6923b040 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd637414740 10 client.16155 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd637414740 10 client.16155 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd637414740 20 client.16155 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd637414740 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb692bbab0 0x55bb692bded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd637414740 4 client.16155 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb691a7dc0 con 0x55bb692bbab0 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb692bbab0 0x55bb692bded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd615ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.767+0000 7fd62dd9b640 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb692bbab0 0x55bb692bded0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ec443b47b6127da7 server_cookie=a7ce9fdbbd8a4144 in_seq=0 out_seq=0 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd615ffb640 1 -- 172.21.15.38:0/3200326209 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb691a7dc0 con 0x55bb692bbab0 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd615ffb640 10 client.16155 handle_command_reply: tid=0 2024-04-22T01:14:46.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 client.16155 shutdown 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 msgr2=0x55bb692b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 0x55bb692b21c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd62003c040 tx=0x7fd62008c900 comp rx=0 tx=0).stop 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 shutdown_connections 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x55bb692bbab0 0x55bb692bded0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb69240ba0 0x55bb692b9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb6923ba30 0x55bb692b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd615ffb640 0 client.16155 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 --2- 172.21.15.38:0/3200326209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb6923b040 0x55bb692b21c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 >> 172.21.15.38:0/3200326209 conn(0x55bb692365b0 msgr2=0x55bb69234fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 shutdown_connections 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 1 -- 172.21.15.38:0/3200326209 wait complete. 2024-04-22T01:14:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd637414740 20 client.16155 trim_cache size 0 max 16384 2024-04-22T01:14:46.775 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:46.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6180a2c10 msgr2=0x7fd6180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6180a2c10 0x7fd6180a50d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd628010c40 tx=0x7fd62800c000 comp rx=0 tx=0).stop 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 msgr2=0x7fd63006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd63006a8d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd620000c00 tx=0x7fd620005dd0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 shutdown_connections 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6180a2c10 0x7fd6180a50d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630102250 0x7fd63019ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630101890 0x7fd63019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 --2- 172.21.15.38:0/2574501178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301006a0 0x7fd63006a8d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.768+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 >> 172.21.15.38:0/2574501178 conn(0x7fd6300fbc30 msgr2=0x7fd630104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.769+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 shutdown_connections 2024-04-22T01:14:46.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.769+0000 7fd63482e640 1 -- 172.21.15.38:0/2574501178 wait complete. 2024-04-22T01:14:46.778 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:46 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/551217905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:14:46.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T01:14:46.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 -- 172.21.15.38:0/3173225236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef8102250 msgr2=0x7f3ef8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 --2- 172.21.15.38:0/3173225236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef8102250 0x7f3ef8108ae0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0009ca0 tx=0x7f3ef002c0f0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 -- 172.21.15.38:0/3173225236 shutdown_connections 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 --2- 172.21.15.38:0/3173225236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef8102250 0x7f3ef8108ae0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 --2- 172.21.15.38:0/3173225236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 0x7f3ef8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 --2- 172.21.15.38:0/3173225236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef81006a0 0x7f3ef8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 -- 172.21.15.38:0/3173225236 >> 172.21.15.38:0/3173225236 conn(0x7f3ef80fbc30 msgr2=0x7f3ef80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 -- 172.21.15.38:0/3173225236 shutdown_connections 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.883+0000 7f3effd27640 1 -- 172.21.15.38:0/3173225236 wait complete. 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 Processor -- start 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 -- start start 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 0x7f3ef8104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 0x7f3ef8102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8102250 0x7f3ef81031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef810c320 con 0x7f3ef81006a0 2024-04-22T01:14:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef810c1a0 con 0x7f3ef8102250 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3effd27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef810c4a0 con 0x7f3ef8101890 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 0x7f3ef8104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 0x7f3ef8104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57544/0 (socket says 172.21.15.38:57544) 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 -- 172.21.15.38:0/1145750831 learned_addr learned my addr 172.21.15.38:0/1145750831 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 0x7f3ef8102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8102250 0x7f3ef81031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 -- 172.21.15.38:0/1145750831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 msgr2=0x7f3ef8102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 0x7f3ef8102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 -- 172.21.15.38:0/1145750831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8102250 msgr2=0x7f3ef81031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8102250 0x7f3ef81031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efdac3640 1 -- 172.21.15.38:0/1145750831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef8103a40 con 0x7f3ef81006a0 2024-04-22T01:14:46.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 0x7f3ef8102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.884+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8102250 0x7f3ef81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.885+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 0x7f3ef8104630 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400d950 tx=0x7f3ef400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.885+0000 7f3eeaffd640 1 -- 172.21.15.38:0/1145750831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4017020 con 0x7f3ef81006a0 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.885+0000 7f3eeaffd640 1 -- 172.21.15.38:0/1145750831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef4009c30 con 0x7f3ef81006a0 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.885+0000 7f3eeaffd640 1 -- 172.21.15.38:0/1145750831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400f720 con 0x7f3ef81006a0 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.885+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef81a1ba0 con 0x7f3ef81006a0 2024-04-22T01:14:46.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.885+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef81a20f0 con 0x7f3ef81006a0 2024-04-22T01:14:46.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.887+0000 7f3eeaffd640 1 -- 172.21.15.38:0/1145750831 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f3ef401a020 con 0x7f3ef81006a0 2024-04-22T01:14:46.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.887+0000 7f3f029ea740 1 Processor -- start 2024-04-22T01:14:46.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 -- start start 2024-04-22T01:14:46.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117040 0x563415117460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x56341511c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x56341511f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56341507c530 con 0x563415117a30 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563415024830 con 0x563415117040 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3f029ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5634150784f0 con 0x56341511cba0 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x56341511c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x56341511c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57548/0 (socket says 172.21.15.38:57548) 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 -- 172.21.15.38:0/3141935880 learned_addr learned my addr 172.21.15.38:0/3141935880 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3eeaffd640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ee40a2b90 0x7f3ee40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x56341511f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117040 0x563415117460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ee40a2b90 0x7f3ee40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3eeaffd640 1 -- 172.21.15.38:0/1145750831 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f3ef40cf490 con 0x7f3ef81006a0 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 -- 172.21.15.38:0/3141935880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 msgr2=0x56341511f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x56341511f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 -- 172.21.15.38:0/3141935880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117040 msgr2=0x563415117460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117040 0x563415117460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.888+0000 7f3efdac3640 1 -- 172.21.15.38:0/3141935880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634150482c0 con 0x563415117a30 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.889+0000 7f3efdac3640 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x56341511c530 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3ef409b620 tx=0x7f3ef4010e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ec00b05092dba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.889+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ee40a2b90 0x7f3ee40a5050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0006010 tx=0x7f3ef00095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.889+0000 7f3ee3fff640 1 -- 172.21.15.38:0/3141935880 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40d2b20 con 0x563415117a30 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.889+0000 7f3ee3fff640 1 -- 172.21.15.38:0/3141935880 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef409c020 con 0x563415117a30 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.889+0000 7f3ee3fff640 1 -- 172.21.15.38:0/3141935880 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40d2d60 con 0x563415117a30 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- 172.21.15.38:0/3141935880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 msgr2=0x56341511c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x56341511c530 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3ef409b620 tx=0x7f3ef4010e30 comp rx=0 tx=0).stop 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- 172.21.15.38:0/3141935880 shutdown_connections 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x56341511f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x56341511c530 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- 172.21.15.38:0/3141935880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117040 0x563415117460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- 172.21.15.38:0/3141935880 >> 172.21.15.38:0/3141935880 conn(0x5634151125b0 msgr2=0x5634151129b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- 172.21.15.38:0/3141935880 shutdown_connections 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- 172.21.15.38:0/3141935880 wait complete. 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- start start 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 Processor -- start 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x563415121250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415121790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x5634151287d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.890+0000 7f3f029ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56341511faf0 con 0x563415117a30 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3f029ea740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563415037f10 con 0x56341511cba0 2024-04-22T01:14:46.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3f029ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56341511fc70 con 0x563415117040 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3f029ea740 10 client.0 _ll_register_callbacks cb 0x563414d8fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415121790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415121790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57562/0 (socket says 172.21.15.38:57562) 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 -- 172.21.15.38:0/1410293932 learned_addr learned my addr 172.21.15.38:0/1410293932 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x5634151287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x563415121250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 -- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 msgr2=0x563415121250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x563415121250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 -- 172.21.15.38:0/1410293932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 msgr2=0x5634151287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x5634151287d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 -- 172.21.15.38:0/1410293932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563415009c80 con 0x563415117a30 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x563415121250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415121790 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3eec011830 tx=0x7f3eec011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc6ddce75002bbb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3ee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3ee17fa640 1 -- 172.21.15.38:0/1410293932 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec00fe10 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3ee17fa640 1 -- 172.21.15.38:0/1410293932 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f3eec005020 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef409c020 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3ee17fa640 1 -- 172.21.15.38:0/1410293932 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3eec024050 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3f029ea740 10 client.16167 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.891+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563414d903f0 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3ee17fa640 1 -- 172.21.15.38:0/1410293932 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef409c020 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3ee17fa640 1 -- 172.21.15.38:0/1410293932 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3eec018030 con 0x563415117a30 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3f029ea740 10 client.16167 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3f029ea740 10 client.16167 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3f029ea740 20 client.16167 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x56341512d160 0x56341512f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3f029ea740 4 client.16167 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563415089a70 con 0x56341512d160 2024-04-22T01:14:46.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x56341512d160 0x56341512f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x56341512d160 0x56341512f600 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ad98833d395c14a server_cookie=bb980c65884e5374 in_seq=0 out_seq=0 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.892+0000 7f3ee17fa640 10 client.16167 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3ee17fa640 1 -- 172.21.15.38:0/1410293932 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563415089a70 con 0x56341512d160 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3ee17fa640 10 client.16167 handle_command_reply: tid=0 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 client.16167 shutdown 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 msgr2=0x563415121790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415121790 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3eec011830 tx=0x7f3eec011cf0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 shutdown_connections 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x56341512d160 0x56341512f600 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x5634151287d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415121790 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1410293932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x563415121250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3ee17fa640 0 client.16167 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 >> 172.21.15.38:0/1410293932 conn(0x5634151125b0 msgr2=0x563415111850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.893+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 shutdown_connections 2024-04-22T01:14:46.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.894+0000 7f3f029ea740 1 -- 172.21.15.38:0/1410293932 wait complete. 2024-04-22T01:14:46.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.894+0000 7f3f029ea740 20 client.16167 trim_cache size 0 max 16384 2024-04-22T01:14:46.902 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:46 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/551217905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:14:46.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 Processor -- start 2024-04-22T01:14:46.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 -- start start 2024-04-22T01:14:46.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 0x563415110fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117a30 0x563415128790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x563415111500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef81ac770 con 0x563415117040 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef81ae2b0 con 0x563415117a30 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3f029ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef81ace40 con 0x56341511cba0 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 0x563415110fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 0x563415110fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57570/0 (socket says 172.21.15.38:57570) 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 -- 172.21.15.38:0/1198335393 learned_addr learned my addr 172.21.15.38:0/1198335393 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117a30 0x563415128790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x563415111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 -- 172.21.15.38:0/1198335393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 msgr2=0x563415111500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x563415111500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 -- 172.21.15.38:0/1198335393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117a30 msgr2=0x563415128790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117a30 0x563415128790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efdac3640 1 -- 172.21.15.38:0/1198335393 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563414e5a7d0 con 0x563415117040 2024-04-22T01:14:46.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117a30 0x563415128790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.907+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x563415111500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3efdac3640 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 0x563415110fc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400dfd0 tx=0x7f3ef409b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1dfd85256a6bc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3ee37fe640 1 -- 172.21.15.38:0/1198335393 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40b4020 con 0x563415117040 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3ee37fe640 1 -- 172.21.15.38:0/1198335393 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef409fcd0 con 0x563415117040 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3ee37fe640 1 -- 172.21.15.38:0/1198335393 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40cf180 con 0x563415117040 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 -- 172.21.15.38:0/1198335393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 msgr2=0x563415110fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 0x563415110fc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400dfd0 tx=0x7f3ef409b920 comp rx=0 tx=0).stop 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 -- 172.21.15.38:0/1198335393 shutdown_connections 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56341511cba0 0x563415111500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563415117a30 0x563415128790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 --2- 172.21.15.38:0/1198335393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117040 0x563415110fc0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 -- 172.21.15.38:0/1198335393 >> 172.21.15.38:0/1198335393 conn(0x5634151125b0 msgr2=0x563415112990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 -- 172.21.15.38:0/1198335393 shutdown_connections 2024-04-22T01:14:46.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.908+0000 7f3f029ea740 1 -- 172.21.15.38:0/1198335393 wait complete. 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 -- start start 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 Processor -- start 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x5634151258c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415125e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x563415126e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef81adbe0 con 0x563415117a30 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef81ad510 con 0x56341511cba0 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef81aa730 con 0x563415117040 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3f029ea740 10 client.0 _ll_register_callbacks cb 0x7f3eec00c7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415125e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415125e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57572/0 (socket says 172.21.15.38:57572) 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 -- 172.21.15.38:0/363834289 learned_addr learned my addr 172.21.15.38:0/363834289 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efdac3640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x5634151258c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x563415126e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 -- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 msgr2=0x5634151258c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x5634151258c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 -- 172.21.15.38:0/363834289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 msgr2=0x563415126e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x563415126e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efd2c2640 1 -- 172.21.15.38:0/363834289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563415127400 con 0x563415117a30 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.909+0000 7f3efe2c4640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x563415126e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3efd2c2640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415125e00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3eec019440 tx=0x7f3eec0198e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1521021bffdd980 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3ee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3ee17fa640 1 -- 172.21.15.38:0/363834289 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec02c070 con 0x563415117a30 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3ee17fa640 1 -- 172.21.15.38:0/363834289 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f3eec011040 con 0x563415117a30 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef409fcd0 con 0x563415117a30 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3ee17fa640 1 -- 172.21.15.38:0/363834289 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3eec038050 con 0x563415117a30 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 10 client.16173 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:46.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5634151286a0 con 0x563415117a30 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3ee17fa640 1 -- 172.21.15.38:0/363834289 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef409fcd0 con 0x563415117a30 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3ee17fa640 1 -- 172.21.15.38:0/363834289 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3eec018030 con 0x563415117a30 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 10 client.16173 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 10 client.16173 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 20 client.16173 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563415193ac0 0x563415195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 4 client.16173 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.910+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563415083dc0 con 0x563415193ac0 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3efdac3640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563415193ac0 0x563415195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3efdac3640 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563415193ac0 0x563415195ee0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd5c66f22566fb17 server_cookie=a963216cb0c4e9cd in_seq=0 out_seq=0 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3ee17fa640 10 client.16173 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3ee17fa640 1 -- 172.21.15.38:0/363834289 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563415083dc0 con 0x563415193ac0 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3ee17fa640 10 client.16173 handle_command_reply: tid=0 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 client.16173 shutdown 2024-04-22T01:14:46.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 msgr2=0x563415125e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415125e00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3eec019440 tx=0x7f3eec0198e0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 shutdown_connections 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x563415193ac0 0x563415195ee0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56341511cba0 0x563415126e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563415117a30 0x563415125e00 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 --2- 172.21.15.38:0/363834289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563415117040 0x5634151258c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3ee17fa640 0 client.16173 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.911+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 >> 172.21.15.38:0/363834289 conn(0x5634151125b0 msgr2=0x563415115810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 shutdown_connections 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3f029ea740 1 -- 172.21.15.38:0/363834289 wait complete. 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3f029ea740 20 client.16173 trim_cache size 0 max 16384 2024-04-22T01:14:46.919 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ee40a2b90 msgr2=0x7f3ee40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ee40a2b90 0x7f3ee40a5050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0006010 tx=0x7f3ef00095c0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 msgr2=0x7f3ef8104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 0x7f3ef8104630 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400d950 tx=0x7f3ef400de10 comp rx=0 tx=0).stop 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 shutdown_connections 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f3ee40a2b90 0x7f3ee40a5050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8102250 0x7f3ef81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8101890 0x7f3ef8102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 --2- 172.21.15.38:0/1145750831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ef81006a0 0x7f3ef8104630 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 >> 172.21.15.38:0/1145750831 conn(0x7f3ef80fbc30 msgr2=0x7f3ef80fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 shutdown_connections 2024-04-22T01:14:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:46.912+0000 7f3effd27640 1 -- 172.21.15.38:0/1145750831 wait complete. 2024-04-22T01:14:46.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T01:14:47.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:46 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/551217905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:14:47.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 -- 172.21.15.38:0/1409483267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 msgr2=0x7fad8c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 --2- 172.21.15.38:0/1409483267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c101d10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fad80009900 tx=0x7fad8002bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 -- 172.21.15.38:0/1409483267 shutdown_connections 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 --2- 172.21.15.38:0/1409483267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad8c102250 0x7fad8c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 --2- 172.21.15.38:0/1409483267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c101d10 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 --2- 172.21.15.38:0/1409483267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad8c1006a0 0x7fad8c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 -- 172.21.15.38:0/1409483267 >> 172.21.15.38:0/1409483267 conn(0x7fad8c0fbc30 msgr2=0x7fad8c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 -- 172.21.15.38:0/1409483267 shutdown_connections 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.026+0000 7fad92975640 1 -- 172.21.15.38:0/1409483267 wait complete. 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 Processor -- start 2024-04-22T01:14:47.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 -- start start 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad8c1006a0 0x7fad8c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad8c102250 0x7fad8c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c10c320 con 0x7fad8c101890 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c10c1a0 con 0x7fad8c102250 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad92975640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c10c4a0 con 0x7fad8c1006a0 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57596/0 (socket says 172.21.15.38:57596) 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1719717347 learned_addr learned my addr 172.21.15.38:0/1719717347 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad90f12640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad8c102250 0x7fad8c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8bfff640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad8c1006a0 0x7fad8c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1719717347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad8c1006a0 msgr2=0x7fad8c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad8c1006a0 0x7fad8c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1719717347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad8c102250 msgr2=0x7fad8c19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad8c102250 0x7fad8c19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.027+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1719717347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8c19f710 con 0x7fad8c101890 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.028+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c198080 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fad80009230 tx=0x7fad80005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.028+0000 7fad897fa640 1 -- 172.21.15.38:0/1719717347 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80041070 con 0x7fad8c101890 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.028+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8c19f9a0 con 0x7fad8c101890 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.028+0000 7fad897fa640 1 -- 172.21.15.38:0/1719717347 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad800044d0 con 0x7fad8c101890 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.028+0000 7fad897fa640 1 -- 172.21.15.38:0/1719717347 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80034440 con 0x7fad8c101890 2024-04-22T01:14:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.028+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad8c19fee0 con 0x7fad8c101890 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad897fa640 1 -- 172.21.15.38:0/1719717347 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fad80044070 con 0x7fad8c101890 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 Processor -- start 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 -- start start 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9a6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6e40 0x558c1d9ab940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9ae440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558c1d907530 con 0x558c1d9a6450 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558c1d8af830 con 0x558c1d9abfb0 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad95634740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c1d9034f0 con 0x558c1d9a6e40 2024-04-22T01:14:47.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad897fa640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fad740a2b90 0x7fad740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad90f12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad90f12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57598/0 (socket says 172.21.15.38:57598) 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad90f12640 1 -- 172.21.15.38:0/3895690126 learned_addr learned my addr 172.21.15.38:0/3895690126 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad90f12640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fad740a2b90 0x7fad740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad897fa640 1 -- 172.21.15.38:0/1719717347 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fad800f1060 con 0x7fad8c101890 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.031+0000 7fad8bfff640 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6e40 0x558c1d9ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 -- 172.21.15.38:0/3895690126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6e40 msgr2=0x558c1d9ab940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6e40 0x558c1d9ab940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 -- 172.21.15.38:0/3895690126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 msgr2=0x558c1d9ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9ae440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 -- 172.21.15.38:0/3895690126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c1d8d32c0 con 0x558c1d9a6450 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9a6870 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fad84011e20 tx=0x7fad8400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c05e5629e186334 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad90f12640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fad740a2b90 0x7fad740a5050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fad8400fc90 tx=0x7fad8401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad727fc640 1 -- 172.21.15.38:0/3895690126 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8401dcb0 con 0x558c1d9a6450 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad727fc640 1 -- 172.21.15.38:0/3895690126 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8401de50 con 0x558c1d9a6450 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.032+0000 7fad727fc640 1 -- 172.21.15.38:0/3895690126 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84004280 con 0x558c1d9a6450 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 -- 172.21.15.38:0/3895690126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 msgr2=0x558c1d9a6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9a6870 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fad84011e20 tx=0x7fad8400f790 comp rx=0 tx=0).stop 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 -- 172.21.15.38:0/3895690126 shutdown_connections 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6e40 0x558c1d9ab940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 --2- 172.21.15.38:0/3895690126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9a6870 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 -- 172.21.15.38:0/3895690126 >> 172.21.15.38:0/3895690126 conn(0x558c1d99d5b0 msgr2=0x558c1d99d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 -- 172.21.15.38:0/3895690126 shutdown_connections 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 -- 172.21.15.38:0/3895690126 wait complete. 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 -- start start 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 Processor -- start 2024-04-22T01:14:47.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9b0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.033+0000 7fad95634740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6e40 0x558c1d9b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad95634740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad95634740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558c1d9aeea0 con 0x558c1d9a6450 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad95634740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558c1d8c2f10 con 0x558c1d9a6e40 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad95634740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c1d9af020 con 0x558c1d9abfb0 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad95634740 10 client.0 _ll_register_callbacks cb 0x558c1d61abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9b0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57604/0 (socket says 172.21.15.38:57604) 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 -- 172.21.15.38:0/680814477 learned_addr learned my addr 172.21.15.38:0/680814477 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6e40 0x558c1d9b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 -- 172.21.15.38:0/680814477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 msgr2=0x558c1d9b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad90f12640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 -- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6e40 msgr2=0x558c1d9b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6e40 0x558c1d9b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 -- 172.21.15.38:0/680814477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c1d894c80 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad90f12640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad8bfff640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9b0630 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00bd90 tx=0x7fad7c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bb59ff3538e74e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad6b7fe640 1 -- 172.21.15.38:0/680814477 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c004220 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad6b7fe640 1 -- 172.21.15.38:0/680814477 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fad7c0043c0 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad8401de50 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.034+0000 7fad6b7fe640 1 -- 172.21.15.38:0/680814477 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad7c024050 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 10 client.16185 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c1d61b3f0 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad6b7fe640 1 -- 172.21.15.38:0/680814477 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8401de50 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad6b7fe640 1 -- 172.21.15.38:0/680814477 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fad7c01b6b0 con 0x558c1d9a6450 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 10 client.16185 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 10 client.16185 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 20 client.16185 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1d9bc540 0x558c1d9bea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 4 client.16185 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c1d914a70 con 0x558c1d9bc540 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.035+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1d9bc540 0x558c1d9bea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.036+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1d9bc540 0x558c1d9bea00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60ffcb30eeec3942 server_cookie=62c6fb996045b2ca in_seq=0 out_seq=0 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.036+0000 7fad6b7fe640 10 client.16185 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.036+0000 7fad6b7fe640 1 -- 172.21.15.38:0/680814477 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c1d914a70 con 0x558c1d9bc540 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.036+0000 7fad6b7fe640 10 client.16185 handle_command_reply: tid=0 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.036+0000 7fad95634740 1 client.16185 shutdown 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 msgr2=0x558c1d9b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9b0630 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00bd90 tx=0x7fad7c0117b0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 shutdown_connections 2024-04-22T01:14:47.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1d9bc540 0x558c1d9bea00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6e40 0x558c1d9b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad6b7fe640 0 client.16185 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 --2- 172.21.15.38:0/680814477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6450 0x558c1d9b0630 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 >> 172.21.15.38:0/680814477 conn(0x558c1d99d5b0 msgr2=0x558c1d99c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 shutdown_connections 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 1 -- 172.21.15.38:0/680814477 wait complete. 2024-04-22T01:14:47.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.037+0000 7fad95634740 20 client.16185 trim_cache size 0 max 16384 2024-04-22T01:14:47.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.050+0000 7fad95634740 1 Processor -- start 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 -- start start 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6450 0x558c1d9a0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9af900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1a8240 con 0x558c1d9a6e40 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1a9d80 con 0x558c1d9abfb0 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad95634740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1a8910 con 0x558c1d9a6450 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57606/0 (socket says 172.21.15.38:57606) 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1583591802 learned_addr learned my addr 172.21.15.38:0/1583591802 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8bfff640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6450 0x558c1d9a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad90f12640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1583591802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6450 msgr2=0x558c1d9a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6450 0x558c1d9a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1583591802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 msgr2=0x558c1d9af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9af900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1583591802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c1d6e57d0 con 0x558c1d9a6e40 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad90f12640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8bfff640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6450 0x558c1d9a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b7b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fad8002c7b0 tx=0x7fad8003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f5413b4f49149 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad71ffb640 1 -- 172.21.15.38:0/1583591802 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad800a6070 con 0x558c1d9a6e40 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad71ffb640 1 -- 172.21.15.38:0/1583591802 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad800bdbf0 con 0x558c1d9a6e40 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.051+0000 7fad71ffb640 1 -- 172.21.15.38:0/1583591802 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8004b040 con 0x558c1d9a6e40 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 -- 172.21.15.38:0/1583591802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 msgr2=0x558c1d9b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b7b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fad8002c7b0 tx=0x7fad8003f040 comp rx=0 tx=0).stop 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 -- 172.21.15.38:0/1583591802 shutdown_connections 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9abfb0 0x558c1d9af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b7b70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 --2- 172.21.15.38:0/1583591802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9a6450 0x558c1d9a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 -- 172.21.15.38:0/1583591802 >> 172.21.15.38:0/1583591802 conn(0x558c1d99d5b0 msgr2=0x558c1d99c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 -- 172.21.15.38:0/1583591802 shutdown_connections 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 -- 172.21.15.38:0/1583591802 wait complete. 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.052+0000 7fad95634740 1 -- start start 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 Processor -- start 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6450 0x558c1d9b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1a96b0 con 0x558c1d9a6e40 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1a8fe0 con 0x558c1d9a6450 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1a6200 con 0x558c1d9abfb0 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 10 client.0 _ll_register_callbacks cb 0x7fad7c005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57614/0 (socket says 172.21.15.38:57614) 2024-04-22T01:14:47.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1044506010 learned_addr learned my addr 172.21.15.38:0/1044506010 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8bfff640 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6450 0x558c1d9b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1044506010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 msgr2=0x558c1d9b6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6450 msgr2=0x558c1d9b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6450 0x558c1d9b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 -- 172.21.15.38:0/1044506010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c1d9b6b10 con 0x558c1d9a6e40 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad8b7fe640 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b5510 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fad800c6320 tx=0x7fad80009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da476375a0a4b41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad6b7fe640 1 -- 172.21.15.38:0/1044506010 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad800d5030 con 0x558c1d9a6e40 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad800bdbf0 con 0x558c1d9a6e40 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.053+0000 7fad6b7fe640 1 -- 172.21.15.38:0/1044506010 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fad800ee8a0 con 0x558c1d9a6e40 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad6b7fe640 1 -- 172.21.15.38:0/1044506010 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad8004b030 con 0x558c1d9a6e40 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 10 client.16191 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:47.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c1d9b7ed0 con 0x558c1d9a6e40 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad6b7fe640 1 -- 172.21.15.38:0/1044506010 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad800bdbf0 con 0x558c1d9a6e40 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad6b7fe640 1 -- 172.21.15.38:0/1044506010 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fad800bd750 con 0x558c1d9a6e40 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 10 client.16191 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 10 client.16191 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 20 client.16191 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1da23170 0x558c1da25590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 4 client.16191 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c1d90edc0 con 0x558c1da23170 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.054+0000 7fad8bfff640 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1da23170 0x558c1da25590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad6b7fe640 10 client.16191 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad8bfff640 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1da23170 0x558c1da25590 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=45de069e4f9211ef server_cookie=e2833b5c2152f713 in_seq=0 out_seq=0 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad6b7fe640 1 -- 172.21.15.38:0/1044506010 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c1d90edc0 con 0x558c1da23170 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad6b7fe640 10 client.16191 handle_command_reply: tid=0 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 client.16191 shutdown 2024-04-22T01:14:47.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 msgr2=0x558c1d9b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b5510 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fad800c6320 tx=0x7fad80009680 comp rx=0 tx=0).stop 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 shutdown_connections 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x558c1da23170 0x558c1da25590 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c1d9abfb0 0x558c1d9b6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558c1d9a6e40 0x558c1d9b5510 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 --2- 172.21.15.38:0/1044506010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558c1d9a6450 0x558c1d9b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad6b7fe640 0 client.16191 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 >> 172.21.15.38:0/1044506010 conn(0x558c1d99d5b0 msgr2=0x558c1d9a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 shutdown_connections 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 1 -- 172.21.15.38:0/1044506010 wait complete. 2024-04-22T01:14:47.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.055+0000 7fad95634740 20 client.16191 trim_cache size 0 max 16384 2024-04-22T01:14:47.062 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:47.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fad740a2b90 msgr2=0x7fad740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fad740a2b90 0x7fad740a5050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fad8400fc90 tx=0x7fad8401e000 comp rx=0 tx=0).stop 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 msgr2=0x7fad8c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c198080 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fad80009230 tx=0x7fad80005d50 comp rx=0 tx=0).stop 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 shutdown_connections 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fad740a2b90 0x7fad740a5050 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad8c102250 0x7fad8c19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad8c101890 0x7fad8c198080 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 --2- 172.21.15.38:0/1719717347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad8c1006a0 0x7fad8c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 >> 172.21.15.38:0/1719717347 conn(0x7fad8c0fbc30 msgr2=0x7fad8c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 shutdown_connections 2024-04-22T01:14:47.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.056+0000 7fad92975640 1 -- 172.21.15.38:0/1719717347 wait complete. 2024-04-22T01:14:47.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T01:14:47.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 -- 172.21.15.38:0/1586312056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a41022c0 msgr2=0x7f27a41046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 --2- 172.21.15.38:0/1586312056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a41022c0 0x7f27a41046b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009900 tx=0x7f27a002bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 -- 172.21.15.38:0/1586312056 shutdown_connections 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 --2- 172.21.15.38:0/1586312056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a4107520 0x7f27a4109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 --2- 172.21.15.38:0/1586312056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 0x7f27a4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 --2- 172.21.15.38:0/1586312056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a41022c0 0x7f27a41046b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 -- 172.21.15.38:0/1586312056 >> 172.21.15.38:0/1586312056 conn(0x7f27a40fbc30 msgr2=0x7f27a40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 -- 172.21.15.38:0/1586312056 shutdown_connections 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.172+0000 7f27ac66b640 1 -- 172.21.15.38:0/1586312056 wait complete. 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 Processor -- start 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 -- start start 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a41022c0 0x7f27a4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 0x7f27a4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 0x7f27a419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410d440 con 0x7f27a4107520 2024-04-22T01:14:47.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410d2c0 con 0x7f27a41022c0 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27ac66b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410d5c0 con 0x7f27a4104bf0 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 0x7f27a419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 0x7f27a419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57630/0 (socket says 172.21.15.38:57630) 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 -- 172.21.15.38:0/2905279963 learned_addr learned my addr 172.21.15.38:0/2905279963 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aa407640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a41022c0 0x7f27a4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27a9c06640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 0x7f27a4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 -- 172.21.15.38:0/2905279963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 msgr2=0x7f27a4198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 0x7f27a4198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 -- 172.21.15.38:0/2905279963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a41022c0 msgr2=0x7f27a4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a41022c0 0x7f27a4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 -- 172.21.15.38:0/2905279963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a419f760 con 0x7f27a4107520 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27a9c06640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 0x7f27a4198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.173+0000 7f27aac08640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 0x7f27a419f0f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f279c00eb10 tx=0x7f279c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.174+0000 7f279b7fe640 1 -- 172.21.15.38:0/2905279963 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c004b20 con 0x7f27a4107520 2024-04-22T01:14:47.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.174+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a419fa50 con 0x7f27a4107520 2024-04-22T01:14:47.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.174+0000 7f279b7fe640 1 -- 172.21.15.38:0/2905279963 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f279c004cc0 con 0x7f27a4107520 2024-04-22T01:14:47.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.174+0000 7f279b7fe640 1 -- 172.21.15.38:0/2905279963 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c010760 con 0x7f27a4107520 2024-04-22T01:14:47.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.174+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a419ffa0 con 0x7f27a4107520 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 Processor -- start 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f279b7fe640 1 -- 172.21.15.38:0/2905279963 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f279c005020 con 0x7f27a4107520 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 -- start start 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579658870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x55657965d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579660440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565795b9530 con 0x55657965dfb0 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556579561830 con 0x556579658e40 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27af254740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5565795b54f0 con 0x556579658450 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27a9c06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579660440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27a9c06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579660440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57644/0 (socket says 172.21.15.38:57644) 2024-04-22T01:14:47.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.177+0000 7f27a9c06640 1 -- 172.21.15.38:0/2139352983 learned_addr learned my addr 172.21.15.38:0/2139352983 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579658870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27aa407640 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x55657965d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f279b7fe640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27900a2b90 0x7f27900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 -- 172.21.15.38:0/2139352983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 msgr2=0x556579658870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579658870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 -- 172.21.15.38:0/2139352983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 msgr2=0x55657965d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x55657965d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 -- 172.21.15.38:0/2139352983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565795852c0 con 0x55657965dfb0 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27aa407640 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x55657965d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27aac08640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27900a2b90 0x7f27900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f27a9c06640 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579660440 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f279400dfd0 tx=0x7f279400d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2382418def0033 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f279b7fe640 1 -- 172.21.15.38:0/2905279963 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f279c0cfa70 con 0x7f27a4107520 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f2798bfe640 1 -- 172.21.15.38:0/2139352983 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794012710 con 0x55657965dfb0 2024-04-22T01:14:47.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f2798bfe640 1 -- 172.21.15.38:0/2139352983 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f279401a070 con 0x55657965dfb0 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.178+0000 7f2798bfe640 1 -- 172.21.15.38:0/2139352983 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794011b30 con 0x55657965dfb0 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 -- 172.21.15.38:0/2139352983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 msgr2=0x556579660440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579660440 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f279400dfd0 tx=0x7f279400d380 comp rx=0 tx=0).stop 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27aac08640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27900a2b90 0x7f27900a5050 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f279c09bc00 tx=0x7f279c0bc040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 -- 172.21.15.38:0/2139352983 shutdown_connections 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579660440 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x55657965d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 --2- 172.21.15.38:0/2139352983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579658870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 -- 172.21.15.38:0/2139352983 >> 172.21.15.38:0/2139352983 conn(0x55657964f5b0 msgr2=0x55657964f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 -- 172.21.15.38:0/2139352983 shutdown_connections 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 -- 172.21.15.38:0/2139352983 wait complete. 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.179+0000 7f27af254740 1 -- start start 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 Processor -- start 2024-04-22T01:14:47.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658450 0x556579662630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658e40 0x556579662b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579669bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556579660f60 con 0x55657965dfb0 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556579574f10 con 0x556579658450 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5565796610e0 con 0x556579658e40 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27af254740 10 client.0 _ll_register_callbacks cb 0x5565792ccbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579669bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579669bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57660/0 (socket says 172.21.15.38:57660) 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 -- 172.21.15.38:0/3071215015 learned_addr learned my addr 172.21.15.38:0/3071215015 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658450 0x556579662630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27aa407640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658e40 0x556579662b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 -- 172.21.15.38:0/3071215015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658e40 msgr2=0x556579662b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658e40 0x556579662b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 -- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658450 msgr2=0x556579662630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658450 0x556579662630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 -- 172.21.15.38:0/3071215015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556579546c80 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f27a9c06640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579669bb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2794001eb0 tx=0x7f2794016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343b9cb69a6d6921 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.180+0000 7f278dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f278dffb640 1 -- 172.21.15.38:0/3071215015 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794025070 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f279401a070 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f278dffb640 1 -- 172.21.15.38:0/3071215015 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f2794005050 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f278dffb640 1 -- 172.21.15.38:0/3071215015 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2794034050 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 10 client.16203 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5565792cd3f0 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f278dffb640 1 -- 172.21.15.38:0/3071215015 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f279401a070 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f278dffb640 1 -- 172.21.15.38:0/3071215015 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2794004d20 con 0x55657965dfb0 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 10 client.16203 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 20 client.16203 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:47.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55657966e150 0x556579670540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 4 client.16203 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.181+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565795c6a70 con 0x55657966e150 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.182+0000 7f27aa407640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55657966e150 0x556579670540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.182+0000 7f27aa407640 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55657966e150 0x556579670540 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ca0eabd7ea4287a server_cookie=765e8e731a3bc7e7 in_seq=0 out_seq=0 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.182+0000 7f278dffb640 10 client.16203 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f278dffb640 1 -- 172.21.15.38:0/3071215015 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565795c6a70 con 0x55657966e150 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f278dffb640 10 client.16203 handle_command_reply: tid=0 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 client.16203 shutdown 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 msgr2=0x556579669bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579669bb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2794001eb0 tx=0x7f2794016040 comp rx=0 tx=0).stop 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 shutdown_connections 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55657966e150 0x556579670540 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55657965dfb0 0x556579669bb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658e40 0x556579662b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f278dffb640 0 client.16203 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:47.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 --2- 172.21.15.38:0/3071215015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658450 0x556579662630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 >> 172.21.15.38:0/3071215015 conn(0x55657964f5b0 msgr2=0x55657964e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 shutdown_connections 2024-04-22T01:14:47.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 1 -- 172.21.15.38:0/3071215015 wait complete. 2024-04-22T01:14:47.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.183+0000 7f27af254740 20 client.16203 trim_cache size 0 max 16384 2024-04-22T01:14:47.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.196+0000 7f27af254740 1 Processor -- start 2024-04-22T01:14:47.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.196+0000 7f27af254740 1 -- start start 2024-04-22T01:14:47.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27af254740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 0x55657964a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27af254740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x556579669b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27af254740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55657965dfb0 0x556579659530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27af254740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41a8310 con 0x556579658450 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27af254740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41a9e50 con 0x556579658e40 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27af254740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41a89e0 con 0x55657965dfb0 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 0x55657964a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 0x55657964a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57662/0 (socket says 172.21.15.38:57662) 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 -- 172.21.15.38:0/1248938983 learned_addr learned my addr 172.21.15.38:0/1248938983 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27aa407640 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x556579669b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 -- 172.21.15.38:0/1248938983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55657965dfb0 msgr2=0x556579659530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55657965dfb0 0x556579659530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 -- 172.21.15.38:0/1248938983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 msgr2=0x556579669b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x556579669b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 -- 172.21.15.38:0/1248938983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565793977d0 con 0x556579658450 2024-04-22T01:14:47.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f27a9c06640 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 0x55657964a6e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2794014380 tx=0x7f2794005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2442f595bf072635 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f278ffff640 1 -- 172.21.15.38:0/1248938983 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794025070 con 0x556579658450 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f278ffff640 1 -- 172.21.15.38:0/1248938983 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f279401a040 con 0x556579658450 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.197+0000 7f278ffff640 1 -- 172.21.15.38:0/1248938983 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279400d8d0 con 0x556579658450 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 -- 172.21.15.38:0/1248938983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 msgr2=0x55657964a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 0x55657964a6e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2794014380 tx=0x7f2794005f40 comp rx=0 tx=0).stop 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 -- 172.21.15.38:0/1248938983 shutdown_connections 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55657965dfb0 0x556579659530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556579658e40 0x556579669b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 --2- 172.21.15.38:0/1248938983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658450 0x55657964a6e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 -- 172.21.15.38:0/1248938983 >> 172.21.15.38:0/1248938983 conn(0x55657964f5b0 msgr2=0x55657964e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 -- 172.21.15.38:0/1248938983 shutdown_connections 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 -- 172.21.15.38:0/1248938983 wait complete. 2024-04-22T01:14:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 -- start start 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 Processor -- start 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.198+0000 7f27af254740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579667190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 0x5565796676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55657965dfb0 0x556579668690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41a9780 con 0x556579658e40 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41a90b0 con 0x55657965dfb0 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41a62d0 con 0x556579658450 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 10 client.0 _ll_register_callbacks cb 0x7f279400df00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 0x5565796676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 0x5565796676d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57664/0 (socket says 172.21.15.38:57664) 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 -- 172.21.15.38:0/205297085 learned_addr learned my addr 172.21.15.38:0/205297085 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27a9c06640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55657965dfb0 0x556579668690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 -- 172.21.15.38:0/205297085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 msgr2=0x556579667190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579667190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 -- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55657965dfb0 msgr2=0x556579668690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55657965dfb0 0x556579668690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 -- 172.21.15.38:0/205297085 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556579668c80 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27a9c06640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55657965dfb0 0x556579668690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27aa407640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 0x5565796676d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f27a002c7b0 tx=0x7f27a002cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e2c6735d1e38e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f278dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f278dffb640 1 -- 172.21.15.38:0/205297085 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a0049070 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f278dffb640 1 -- 172.21.15.38:0/205297085 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f27a0045dc0 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f279401a040 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f278dffb640 1 -- 172.21.15.38:0/205297085 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27a0055050 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 10 client.16209 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.199+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5565796d4590 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f278dffb640 1 -- 172.21.15.38:0/205297085 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f279401a040 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f278dffb640 1 -- 172.21.15.38:0/205297085 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f27a0005280 con 0x556579658e40 2024-04-22T01:14:47.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27af254740 10 client.16209 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27af254740 10 client.16209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27af254740 20 client.16209 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27af254740 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5565796d4e50 0x5565796d7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27af254740 4 client.16209 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565795c0dc0 con 0x5565796d4e50 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.200+0000 7f27a9c06640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5565796d4e50 0x5565796d7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27a9c06640 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5565796d4e50 0x5565796d7260 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bf0b757816c65f server_cookie=5d4cf976dbe967b3 in_seq=0 out_seq=0 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f278dffb640 10 client.16209 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f278dffb640 1 -- 172.21.15.38:0/205297085 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565795c0dc0 con 0x5565796d4e50 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f278dffb640 10 client.16209 handle_command_reply: tid=0 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 client.16209 shutdown 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 msgr2=0x5565796676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 0x5565796676d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f27a002c7b0 tx=0x7f27a002cba0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 shutdown_connections 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x5565796d4e50 0x5565796d7260 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55657965dfb0 0x556579668690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f278dffb640 0 client.16209 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556579658e40 0x5565796676d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 --2- 172.21.15.38:0/205297085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556579658450 0x556579667190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 >> 172.21.15.38:0/205297085 conn(0x55657964f5b0 msgr2=0x556579652810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 shutdown_connections 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 1 -- 172.21.15.38:0/205297085 wait complete. 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27af254740 20 client.16209 trim_cache size 0 max 16384 2024-04-22T01:14:47.208 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:14:47.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27900a2b90 msgr2=0x7f27900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27ac66b640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27900a2b90 0x7f27900a5050 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f279c09bc00 tx=0x7f279c0bc040 comp rx=0 tx=0).stop 2024-04-22T01:14:47.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 msgr2=0x7f27a419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:47.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.201+0000 7f27ac66b640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 0x7f27a419f0f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f279c00eb10 tx=0x7f279c00c2d0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 shutdown_connections 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27900a2b90 0x7f27900a5050 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27a4107520 0x7f27a419f0f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a4104bf0 0x7f27a4198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 --2- 172.21.15.38:0/2905279963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a41022c0 0x7f27a4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 >> 172.21.15.38:0/2905279963 conn(0x7f27a40fbc30 msgr2=0x7f27a4102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 shutdown_connections 2024-04-22T01:14:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:47.202+0000 7f27ac66b640 1 -- 172.21.15.38:0/2905279963 wait complete. 2024-04-22T01:14:47.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:47 smithi111 ceph-mon[31057]: pgmap v428: 129 pgs: 129 active+clean; 1.6 GiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 127 MiB/s wr, 55 op/s 2024-04-22T01:14:47.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:47 smithi038 ceph-mon[28043]: pgmap v428: 129 pgs: 129 active+clean; 1.6 GiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 127 MiB/s wr, 55 op/s 2024-04-22T01:14:48.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:47 smithi203 ceph-mon[31825]: pgmap v428: 129 pgs: 129 active+clean; 1.6 GiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 127 MiB/s wr, 55 op/s 2024-04-22T01:14:48.213 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T01:14:48.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:14:48.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 -- 172.21.15.38:0/718673428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638103350 msgr2=0x7fe638109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/718673428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638103350 0x7fe638109be0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe630009900 tx=0x7fe63002bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 -- 172.21.15.38:0/718673428 shutdown_connections 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/718673428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638103350 0x7fe638109be0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/718673428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638102990 0x7fe638102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/718673428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6381017a0 0x7fe638101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 -- 172.21.15.38:0/718673428 >> 172.21.15.38:0/718673428 conn(0x7fe6380fcd10 msgr2=0x7fe6380ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 -- 172.21.15.38:0/718673428 shutdown_connections 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.333+0000 7fe63e0fb640 1 -- 172.21.15.38:0/718673428 wait complete. 2024-04-22T01:14:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 Processor -- start 2024-04-22T01:14:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 -- start start 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6381017a0 0x7fe638197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 0x7fe638197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638103350 0x7fe63819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe63810d420 con 0x7fe638102990 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe63810d2a0 con 0x7fe638103350 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.334+0000 7fe63e0fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe63810d5a0 con 0x7fe6381017a0 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 0x7fe638197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 0x7fe638197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57688/0 (socket says 172.21.15.38:57688) 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 -- 172.21.15.38:0/2875313968 learned_addr learned my addr 172.21.15.38:0/2875313968 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe637fff640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638103350 0x7fe63819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe6377fe640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6381017a0 0x7fe638197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 -- 172.21.15.38:0/2875313968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6381017a0 msgr2=0x7fe638197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6381017a0 0x7fe638197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 -- 172.21.15.38:0/2875313968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638103350 msgr2=0x7fe63819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638103350 0x7fe63819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 -- 172.21.15.38:0/2875313968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe63819f680 con 0x7fe638102990 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe637fff640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638103350 0x7fe63819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe636ffd640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 0x7fe638197fa0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00d950 tx=0x7fe62c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe634ff9640 1 -- 172.21.15.38:0/2875313968 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c017030 con 0x7fe638102990 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe634ff9640 1 -- 172.21.15.38:0/2875313968 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c004020 con 0x7fe638102990 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe63819f970 con 0x7fe638102990 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe634ff9640 1 -- 172.21.15.38:0/2875313968 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0047a0 con 0x7fe638102990 2024-04-22T01:14:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.335+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe63819fec0 con 0x7fe638102990 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe634ff9640 1 -- 172.21.15.38:0/2875313968 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe62c0041d0 con 0x7fe638102990 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 Processor -- start 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 -- start start 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c0492dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0493390 0x55a3c0497e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 0x55a3c049a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a3c03f7170 con 0x55a3c0498500 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a3c039f7c0 con 0x55a3c04929a0 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe640dc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a3c03f3130 con 0x55a3c0493390 2024-04-22T01:14:48.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 0x55a3c049a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe634ff9640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6200a2b90 0x7fe6200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 0x55a3c049a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57704/0 (socket says 172.21.15.38:57704) 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.338+0000 7fe636ffd640 1 -- 172.21.15.38:0/3867238995 learned_addr learned my addr 172.21.15.38:0/3867238995 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe6377fe640 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0493390 0x55a3c0497e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe637fff640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6200a2b90 0x7fe6200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe637fff640 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c0492dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe634ff9640 1 -- 172.21.15.38:0/2875313968 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe62c0ceeb0 con 0x7fe638102990 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe636ffd640 1 -- 172.21.15.38:0/3867238995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0493390 msgr2=0x55a3c0497e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe636ffd640 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0493390 0x55a3c0497e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe636ffd640 1 -- 172.21.15.38:0/3867238995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 msgr2=0x55a3c0492dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe636ffd640 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c0492dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe636ffd640 1 -- 172.21.15.38:0/3867238995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c03b0e60 con 0x55a3c0498500 2024-04-22T01:14:48.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe636ffd640 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 0x55a3c049a990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0051b0 tx=0x7fe62c09af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36007275d6ec1a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe637fff640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6200a2b90 0x7fe6200a5050 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe630006010 tx=0x7fe630005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe61dffb640 1 -- 172.21.15.38:0/3867238995 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c09b340 con 0x55a3c0498500 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe61dffb640 1 -- 172.21.15.38:0/3867238995 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c09ba40 con 0x55a3c0498500 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.339+0000 7fe61dffb640 1 -- 172.21.15.38:0/3867238995 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c09a790 con 0x55a3c0498500 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 -- 172.21.15.38:0/3867238995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 msgr2=0x55a3c049a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 0x55a3c049a990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0051b0 tx=0x7fe62c09af40 comp rx=0 tx=0).stop 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 -- 172.21.15.38:0/3867238995 shutdown_connections 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0498500 0x55a3c049a990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0493390 0x55a3c0497e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3867238995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c0492dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 -- 172.21.15.38:0/3867238995 >> 172.21.15.38:0/3867238995 conn(0x55a3c045a4f0 msgr2=0x55a3c048e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 -- 172.21.15.38:0/3867238995 shutdown_connections 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 -- 172.21.15.38:0/3867238995 wait complete. 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.340+0000 7fe640dc1740 1 -- start start 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 Processor -- start 2024-04-22T01:14:48.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c049ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c049d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a3c049b3f0 con 0x55a3c0493390 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a3c03b2d40 con 0x55a3c04929a0 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a3c049b570 con 0x55a3c0498500 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe640dc1740 10 client.0 _ll_register_callbacks cb 0x55a3c03ee820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c049d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c049d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57720/0 (socket says 172.21.15.38:57720) 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 -- 172.21.15.38:0/1717450484 learned_addr learned my addr 172.21.15.38:0/1717450484 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe637fff640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c049ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 -- 172.21.15.38:0/1717450484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 msgr2=0x55a3c04a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 -- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 msgr2=0x55a3c049ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c049ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 -- 172.21.15.38:0/1717450484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c01d37e0 con 0x55a3c0493390 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c049ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.341+0000 7fe636ffd640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c049d390 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0977a0 tx=0x7fe62c099640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b18f033983de61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe616ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe616ffd640 1 -- 172.21.15.38:0/1717450484 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0975e0 con 0x55a3c0493390 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe616ffd640 1 -- 172.21.15.38:0/1717450484 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe62c09a790 con 0x55a3c0493390 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe62c09ba40 con 0x55a3c0493390 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe616ffd640 1 -- 172.21.15.38:0/1717450484 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe62c09b340 con 0x55a3c0493390 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 10 client.16221 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3c0233f70 con 0x55a3c0493390 2024-04-22T01:14:48.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe616ffd640 1 -- 172.21.15.38:0/1717450484 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62c09ba40 con 0x55a3c0493390 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe616ffd640 1 -- 172.21.15.38:0/1717450484 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe62c09a200 con 0x55a3c0493390 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 10 client.16221 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 20 client.16221 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c04a8530 0x55a3c04aa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 4 client.16221 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.342+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3c03cb450 con 0x55a3c04a8530 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.343+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c04a8530 0x55a3c04aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.343+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c04a8530 0x55a3c04aa920 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75b31b952ce53ab1 server_cookie=c9db60b01be7626b in_seq=0 out_seq=0 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.343+0000 7fe616ffd640 10 client.16221 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe616ffd640 1 -- 172.21.15.38:0/1717450484 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a3c03cb450 con 0x55a3c04a8530 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe616ffd640 10 client.16221 handle_command_reply: tid=0 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 client.16221 shutdown 2024-04-22T01:14:48.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 msgr2=0x55a3c049d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c049d390 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0977a0 tx=0x7fe62c099640 comp rx=0 tx=0).stop 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 shutdown_connections 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c04a8530 0x55a3c04aa920 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe616ffd640 0 client.16221 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c049d390 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1717450484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c049ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 >> 172.21.15.38:0/1717450484 conn(0x55a3c045a4f0 msgr2=0x55a3c048d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 shutdown_connections 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 1 -- 172.21.15.38:0/1717450484 wait complete. 2024-04-22T01:14:48.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.344+0000 7fe640dc1740 20 client.16221 trim_cache size 0 max 16384 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 Processor -- start 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 -- start start 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 0x55a3c0491550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c0493390 0x55a3c04a4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c0491a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381aa4f0 con 0x55a3c04929a0 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381ac030 con 0x55a3c0493390 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe640dc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381aabc0 con 0x55a3c0498500 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 0x55a3c0491550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 0x55a3c0491550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57730/0 (socket says 172.21.15.38:57730) 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 -- 172.21.15.38:0/3877550417 learned_addr learned my addr 172.21.15.38:0/3877550417 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe637fff640 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c0491a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 -- 172.21.15.38:0/3877550417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 msgr2=0x55a3c0491a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c0491a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 -- 172.21.15.38:0/3877550417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c0493390 msgr2=0x55a3c04a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c0493390 0x55a3c04a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.362+0000 7fe6377fe640 1 -- 172.21.15.38:0/3877550417 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c00cb170 con 0x55a3c04929a0 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe6377fe640 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 0x55a3c0491550 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe6381027f0 tx=0x7fe62800a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228caaf8f8277db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe61d7fa640 1 -- 172.21.15.38:0/3877550417 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62801e020 con 0x55a3c04929a0 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe61d7fa640 1 -- 172.21.15.38:0/3877550417 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62801a810 con 0x55a3c04929a0 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe61d7fa640 1 -- 172.21.15.38:0/3877550417 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628012b90 con 0x55a3c04929a0 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 -- 172.21.15.38:0/3877550417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 msgr2=0x55a3c0491550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 0x55a3c0491550 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe6381027f0 tx=0x7fe62800a6d0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 -- 172.21.15.38:0/3877550417 shutdown_connections 2024-04-22T01:14:48.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c0491a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c0493390 0x55a3c04a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 --2- 172.21.15.38:0/3877550417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c04929a0 0x55a3c0491550 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 -- 172.21.15.38:0/3877550417 >> 172.21.15.38:0/3877550417 conn(0x55a3c045a4f0 msgr2=0x55a3c04948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 -- 172.21.15.38:0/3877550417 shutdown_connections 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.363+0000 7fe640dc1740 1 -- 172.21.15.38:0/3877550417 wait complete. 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 -- start start 2024-04-22T01:14:48.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 Processor -- start 2024-04-22T01:14:48.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c04a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c04a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381ab960 con 0x55a3c0493390 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381ab290 con 0x55a3c04929a0 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6381a84b0 con 0x55a3c0498500 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe640dc1740 10 client.0 _ll_register_callbacks cb 0x7fe62800a570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c04a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe636ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c04a22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57740/0 (socket says 172.21.15.38:57740) 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe636ffd640 1 -- 172.21.15.38:0/1407783436 learned_addr learned my addr 172.21.15.38:0/1407783436 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.364+0000 7fe637fff640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe636ffd640 1 -- 172.21.15.38:0/1407783436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 msgr2=0x55a3c04a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c04a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe636ffd640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe636ffd640 1 -- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 msgr2=0x55a3c04a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe636ffd640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c04a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe636ffd640 1 -- 172.21.15.38:0/1407783436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c04a3810 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe636ffd640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c04a22d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009fd0 tx=0x7fe62c00b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18bc69b390b6cb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe616ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe616ffd640 1 -- 172.21.15.38:0/1407783436 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0975e0 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe616ffd640 1 -- 172.21.15.38:0/1407783436 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe62c0bb0b0 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe62801a810 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe616ffd640 1 -- 172.21.15.38:0/1407783436 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe62c09b110 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 10 client.16227 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3c050ea60 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe616ffd640 1 -- 172.21.15.38:0/1407783436 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe62801a810 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe616ffd640 1 -- 172.21.15.38:0/1407783436 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe62c09a790 con 0x55a3c0493390 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 10 client.16227 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 20 client.16227 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c050f2d0 0x55a3c05116f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.365+0000 7fe640dc1740 4 client.16227 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.366+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a3c04046b0 con 0x55a3c050f2d0 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.366+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c050f2d0 0x55a3c05116f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.366+0000 7fe616ffd640 10 client.16227 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.366+0000 7fe6377fe640 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c050f2d0 0x55a3c05116f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8de9af261a2f7018 server_cookie=95b871a5a97b311b in_seq=0 out_seq=0 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.367+0000 7fe616ffd640 1 -- 172.21.15.38:0/1407783436 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a3c04046b0 con 0x55a3c050f2d0 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.367+0000 7fe616ffd640 10 client.16227 handle_command_reply: tid=0 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.367+0000 7fe640dc1740 1 client.16227 shutdown 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.367+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 msgr2=0x55a3c04a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.367+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c04a22d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009fd0 tx=0x7fe62c00b510 comp rx=0 tx=0).stop 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 shutdown_connections 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55a3c050f2d0 0x55a3c05116f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c0498500 0x55a3c04a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3c0493390 0x55a3c04a22d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 --2- 172.21.15.38:0/1407783436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3c04929a0 0x55a3c04a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 >> 172.21.15.38:0/1407783436 conn(0x55a3c045a4f0 msgr2=0x55a3c048cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe616ffd640 0 client.16227 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 shutdown_connections 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 1 -- 172.21.15.38:0/1407783436 wait complete. 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe640dc1740 20 client.16227 trim_cache size 0 max 16384 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "bbbb38b0-7ea3-4dbc-869b-faf0fe4ee33a", 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T01:14:48.373 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:14:48.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6200a2b90 msgr2=0x7fe6200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6200a2b90 0x7fe6200a5050 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe630006010 tx=0x7fe630005a20 comp rx=0 tx=0).stop 2024-04-22T01:14:48.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 msgr2=0x7fe638197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 0x7fe638197fa0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00d950 tx=0x7fe62c00de10 comp rx=0 tx=0).stop 2024-04-22T01:14:48.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 shutdown_connections 2024-04-22T01:14:48.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe6200a2b90 0x7fe6200a5050 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe638103350 0x7fe63819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe638102990 0x7fe638197fa0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 --2- 172.21.15.38:0/2875313968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6381017a0 0x7fe638197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.368+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 >> 172.21.15.38:0/2875313968 conn(0x7fe6380fcd10 msgr2=0x7fe638105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.369+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 shutdown_connections 2024-04-22T01:14:48.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.369+0000 7fe63e0fb640 1 -- 172.21.15.38:0/2875313968 wait complete. 2024-04-22T01:14:48.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:14:48.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:14:48.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 -- 172.21.15.38:0/523255832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 msgr2=0x7ff44c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 --2- 172.21.15.38:0/523255832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c108ae0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff444009900 tx=0x7ff44402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:48.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 -- 172.21.15.38:0/523255832 shutdown_connections 2024-04-22T01:14:48.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 --2- 172.21.15.38:0/523255832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c108ae0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 --2- 172.21.15.38:0/523255832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff44c101890 0x7ff44c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 --2- 172.21.15.38:0/523255832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44c1006a0 0x7ff44c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 -- 172.21.15.38:0/523255832 >> 172.21.15.38:0/523255832 conn(0x7ff44c0fbc30 msgr2=0x7ff44c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 -- 172.21.15.38:0/523255832 shutdown_connections 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 -- 172.21.15.38:0/523255832 wait complete. 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 Processor -- start 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.484+0000 7ff453572640 1 -- start start 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44c1006a0 0x7ff44c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff44c101890 0x7ff44c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c10d4a0 con 0x7ff44c102250 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c10d320 con 0x7ff44c1006a0 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c10d620 con 0x7ff44c101890 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57764/0 (socket says 172.21.15.38:57764) 2024-04-22T01:14:48.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 -- 172.21.15.38:0/19137649 learned_addr learned my addr 172.21.15.38:0/19137649 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff45130e640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44c1006a0 0x7ff44c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff450b0d640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff44c101890 0x7ff44c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 -- 172.21.15.38:0/19137649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff44c101890 msgr2=0x7ff44c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff44c101890 0x7ff44c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 -- 172.21.15.38:0/19137649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44c1006a0 msgr2=0x7ff44c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44c1006a0 0x7ff44c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 -- 172.21.15.38:0/19137649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c19f6e0 con 0x7ff44c102250 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff451b0f640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c19f070 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff444009230 tx=0x7ff444004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff44a7fc640 1 -- 172.21.15.38:0/19137649 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444041070 con 0x7ff44c102250 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff44a7fc640 1 -- 172.21.15.38:0/19137649 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4440054e0 con 0x7ff44c102250 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44c19f970 con 0x7ff44c102250 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.485+0000 7ff44a7fc640 1 -- 172.21.15.38:0/19137649 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4440346a0 con 0x7ff44c102250 2024-04-22T01:14:48.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.486+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff44c19fe60 con 0x7ff44c102250 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff44a7fc640 1 -- 172.21.15.38:0/19137649 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7ff444044070 con 0x7ff44c102250 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 Processor -- start 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 -- start start 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1ad440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1aff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cd7f10d560 con 0x55cd7f1a7f50 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cd7f0b5730 con 0x55cd7f1adab0 2024-04-22T01:14:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff456231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cd7f109520 con 0x55cd7f1a8940 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff44a7fc640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff4340a2b90 0x7ff4340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff450b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff450b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57768/0 (socket says 172.21.15.38:57768) 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff450b0d640 1 -- 172.21.15.38:0/3679900714 learned_addr learned my addr 172.21.15.38:0/3679900714 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff45130e640 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.488+0000 7ff451b0f640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff4340a2b90 0x7ff4340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff44a7fc640 1 -- 172.21.15.38:0/19137649 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff4440f1370 con 0x7ff44c102250 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff45130e640 1 -- 172.21.15.38:0/3679900714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 msgr2=0x55cd7f1aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff45130e640 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff45130e640 1 -- 172.21.15.38:0/3679900714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 msgr2=0x55cd7f1a8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff45130e640 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff45130e640 1 -- 172.21.15.38:0/3679900714 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd7eeeb7d0 con 0x55cd7f1a8940 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff451b0f640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff4340a2b90 0x7ff4340a5050 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff44402cec0 tx=0x7ff444088040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff45130e640 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1ad440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009eb0 tx=0x7ff43c00c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=868ed853fdffd9d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff4337fe640 1 -- 172.21.15.38:0/3679900714 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c01a070 con 0x55cd7f1a8940 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff4337fe640 1 -- 172.21.15.38:0/3679900714 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff43c0158f0 con 0x55cd7f1a8940 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.489+0000 7ff4337fe640 1 -- 172.21.15.38:0/3679900714 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c00a960 con 0x55cd7f1a8940 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 -- 172.21.15.38:0/3679900714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 msgr2=0x55cd7f1ad440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1ad440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009eb0 tx=0x7ff43c00c7f0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 -- 172.21.15.38:0/3679900714 shutdown_connections 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1ad440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 --2- 172.21.15.38:0/3679900714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 -- 172.21.15.38:0/3679900714 >> 172.21.15.38:0/3679900714 conn(0x55cd7f1a34c0 msgr2=0x55cd7f1a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 -- 172.21.15.38:0/3679900714 shutdown_connections 2024-04-22T01:14:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 -- 172.21.15.38:0/3679900714 wait complete. 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 -- start start 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.490+0000 7ff456231740 1 Processor -- start 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1b2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f1b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1adab0 0x55cd7f1b9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cd7f1b0a60 con 0x55cd7f1a8940 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cd7f0c8e00 con 0x55cd7f1a7f50 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cd7f1b0be0 con 0x55cd7f1adab0 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff456231740 10 client.0 _ll_register_callbacks cb 0x55cd7f103eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f1b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f1b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57784/0 (socket says 172.21.15.38:57784) 2024-04-22T01:14:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 -- 172.21.15.38:0/2820983886 learned_addr learned my addr 172.21.15.38:0/2820983886 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff450b0d640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff450b0d640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1adab0 0x55cd7f1b9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 -- 172.21.15.38:0/2820983886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1adab0 msgr2=0x55cd7f1b9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1adab0 0x55cd7f1b9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 -- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1a7f50 msgr2=0x55cd7f1b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1b2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 -- 172.21.15.38:0/2820983886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd7ee213f0 con 0x55cd7f1a8940 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff450b0d640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff45130e640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f1b2a70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff43c004b90 tx=0x7ff43c015640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8556fb5a8dc64bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff430ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.491+0000 7ff430ff9640 1 -- 172.21.15.38:0/2820983886 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c01a040 con 0x55cd7f1a8940 2024-04-22T01:14:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff430ff9640 1 -- 172.21.15.38:0/2820983886 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff43c00a6e0 con 0x55cd7f1a8940 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff43c0158f0 con 0x55cd7f1a8940 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff430ff9640 1 -- 172.21.15.38:0/2820983886 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff43c009b30 con 0x55cd7f1a8940 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 10 client.16239 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cd7ede10f0 con 0x55cd7f1a8940 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff430ff9640 1 -- 172.21.15.38:0/2820983886 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff43c0158f0 con 0x55cd7f1a8940 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff430ff9640 1 -- 172.21.15.38:0/2820983886 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff43c0208c0 con 0x55cd7f1a8940 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 10 client.16239 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 20 client.16239 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f1be0b0 0x55cd7f1c04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 4 client.16239 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd7f11aaa0 con 0x55cd7f1be0b0 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.492+0000 7ff450b0d640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f1be0b0 0x55cd7f1c04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff430ff9640 10 client.16239 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff450b0d640 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f1be0b0 0x55cd7f1c04a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f86ace0076956d server_cookie=d701e7c7202b418e in_seq=0 out_seq=0 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff430ff9640 1 -- 172.21.15.38:0/2820983886 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd7f11aaa0 con 0x55cd7f1be0b0 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff430ff9640 10 client.16239 handle_command_reply: tid=0 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff456231740 1 client.16239 shutdown 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 msgr2=0x55cd7f1b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.493+0000 7ff456231740 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f1b2a70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff43c004b90 tx=0x7ff43c015640 comp rx=0 tx=0).stop 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 shutdown_connections 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f1be0b0 0x55cd7f1c04a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1adab0 0x55cd7f1b9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff430ff9640 0 client.16239 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f1b2a70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 --2- 172.21.15.38:0/2820983886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 >> 172.21.15.38:0/2820983886 conn(0x55cd7f1a34c0 msgr2=0x55cd7f1a26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 shutdown_connections 2024-04-22T01:14:48.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 1 -- 172.21.15.38:0/2820983886 wait complete. 2024-04-22T01:14:48.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.494+0000 7ff456231740 20 client.16239 trim_cache size 0 max 16384 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.511+0000 7ff456231740 1 Processor -- start 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.511+0000 7ff456231740 1 -- start start 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.511+0000 7ff456231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.511+0000 7ff456231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1a2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.511+0000 7ff456231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff456231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c1aa4a0 con 0x55cd7f1a7f50 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff456231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c1abfe0 con 0x55cd7f1adab0 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff456231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c1aab70 con 0x55cd7f1a8940 2024-04-22T01:14:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57798/0 (socket says 172.21.15.38:57798) 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 -- 172.21.15.38:0/3911020664 learned_addr learned my addr 172.21.15.38:0/3911020664 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff45130e640 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 -- 172.21.15.38:0/3911020664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 msgr2=0x55cd7f1a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 -- 172.21.15.38:0/3911020664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 msgr2=0x55cd7f1a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 -- 172.21.15.38:0/3911020664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd7f1a4230 con 0x55cd7f1a7f50 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a1e50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff44000c010 tx=0x7ff44000b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b87a57bbf083c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff432ffd640 1 -- 172.21.15.38:0/3911020664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440005220 con 0x55cd7f1a7f50 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff432ffd640 1 -- 172.21.15.38:0/3911020664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff440016070 con 0x55cd7f1a7f50 2024-04-22T01:14:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.512+0000 7ff432ffd640 1 -- 172.21.15.38:0/3911020664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440012a20 con 0x55cd7f1a7f50 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 -- 172.21.15.38:0/3911020664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 msgr2=0x55cd7f1a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a1e50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff44000c010 tx=0x7ff44000b2c0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 -- 172.21.15.38:0/3911020664 shutdown_connections 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f1a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a8940 0x55cd7f1a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 --2- 172.21.15.38:0/3911020664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a7f50 0x55cd7f1a1e50 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 -- 172.21.15.38:0/3911020664 >> 172.21.15.38:0/3911020664 conn(0x55cd7f1a34c0 msgr2=0x55cd7f1a38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 -- 172.21.15.38:0/3911020664 shutdown_connections 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 -- 172.21.15.38:0/3911020664 wait complete. 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 -- start start 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.513+0000 7ff456231740 1 Processor -- start 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a7f50 0x55cd7f2203d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f220910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f2278e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c1ab910 con 0x55cd7f1a8940 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c1ab240 con 0x55cd7f1adab0 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c1a8460 con 0x55cd7f1a7f50 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff456231740 10 client.0 _ll_register_callbacks cb 0x7ff43c018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f220910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f220910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57802/0 (socket says 172.21.15.38:57802) 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 -- 172.21.15.38:0/3578431090 learned_addr learned my addr 172.21.15.38:0/3578431090 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a7f50 0x55cd7f2203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f2278e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 -- 172.21.15.38:0/3578431090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a7f50 msgr2=0x55cd7f2203d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a7f50 0x55cd7f2203d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 -- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 msgr2=0x55cd7f2278e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f2278e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 -- 172.21.15.38:0/3578431090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd7f1a2e10 con 0x55cd7f1a8940 2024-04-22T01:14:48.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a7f50 0x55cd7f2203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff45130e640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f220910 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55cd7f1a4400 tx=0x7ff43c007110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bebe293a74a907 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff430ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.514+0000 7ff430ff9640 1 -- 172.21.15.38:0/3578431090 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c01a040 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff430ff9640 1 -- 172.21.15.38:0/3578431090 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7ff43c0156b0 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff440016070 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff430ff9640 1 -- 172.21.15.38:0/3578431090 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff43c03b050 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 10 client.16245 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cd7f1b2de0 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff430ff9640 1 -- 172.21.15.38:0/3578431090 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff440016070 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff430ff9640 1 -- 172.21.15.38:0/3578431090 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff43c030020 con 0x55cd7f1a8940 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 10 client.16245 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 20 client.16245 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:48.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f229780 0x55cd7f22bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 4 client.16245 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cd7f114df0 con 0x55cd7f229780 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.515+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f229780 0x55cd7f22bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff430ff9640 10 client.16245 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff450b0d640 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f229780 0x55cd7f22bba0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62fbe211017fb9b1 server_cookie=8c430fb25c6d3c84 in_seq=0 out_seq=0 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff430ff9640 1 -- 172.21.15.38:0/3578431090 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cd7f114df0 con 0x55cd7f229780 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff430ff9640 10 client.16245 handle_command_reply: tid=0 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 client.16245 shutdown 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 msgr2=0x55cd7f220910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f220910 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55cd7f1a4400 tx=0x7ff43c007110 comp rx=0 tx=0).stop 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 shutdown_connections 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cd7f229780 0x55cd7f22bba0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cd7f1adab0 0x55cd7f2278e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd7f1a8940 0x55cd7f220910 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff430ff9640 0 client.16245 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 --2- 172.21.15.38:0/3578431090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cd7f1a7f50 0x55cd7f2203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 >> 172.21.15.38:0/3578431090 conn(0x55cd7f1a34c0 msgr2=0x55cd7f1a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 shutdown_connections 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 1 -- 172.21.15.38:0/3578431090 wait complete. 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.516+0000 7ff456231740 20 client.16245 trim_cache size 0 max 16384 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:14:48.523 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: "bbbb38b0-7ea3-4dbc-869b-faf0fe4ee33a": { 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "bbbb38b0-7ea3-4dbc-869b-faf0fe4ee33a", 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:14:48.524 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff4340a2b90 msgr2=0x7ff4340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff4340a2b90 0x7ff4340a5050 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff44402cec0 tx=0x7ff444088040 comp rx=0 tx=0).stop 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 msgr2=0x7ff44c19f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c19f070 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff444009230 tx=0x7ff444004040 comp rx=0 tx=0).stop 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 shutdown_connections 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7ff4340a2b90 0x7ff4340a5050 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff44c102250 0x7ff44c19f070 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff44c101890 0x7ff44c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 --2- 172.21.15.38:0/19137649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44c1006a0 0x7ff44c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:48.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 >> 172.21.15.38:0/19137649 conn(0x7ff44c0fbc30 msgr2=0x7ff44c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:48.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 shutdown_connections 2024-04-22T01:14:48.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:14:48.517+0000 7ff453572640 1 -- 172.21.15.38:0/19137649 wait complete. 2024-04-22T01:14:48.528 INFO:tasks.cephfs.filesystem:scrub status for tag:bbbb38b0-7ea3-4dbc-869b-faf0fe4ee33a - {'path': '/', 'tag': 'bbbb38b0-7ea3-4dbc-869b-faf0fe4ee33a', 'options': 'recursive,force'} 2024-04-22T01:14:48.828 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:48.828 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 re-readers = 8865049.00 kB/sec 2024-04-22T01:14:48.829 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 re-readers = 8847180.08 kB/sec 2024-04-22T01:14:48.829 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 8865049.00 kB/sec 2024-04-22T01:14:48.829 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 8865049.00 kB/sec 2024-04-22T01:14:48.829 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 8865049.00 kB/sec 2024-04-22T01:14:48.829 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 1048576.00 kB 2024-04-22T01:14:48.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:48 smithi111 ceph-mon[31057]: scrub queued for path: / 2024-04-22T01:14:48.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:48 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:48.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:48 smithi111 ceph-mon[31057]: scrub summary: active paths [/] 2024-04-22T01:14:48.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:48 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:48.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:48 smithi038 ceph-mon[28043]: scrub queued for path: / 2024-04-22T01:14:48.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:48 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:48.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:48 smithi038 ceph-mon[28043]: scrub summary: active paths [/] 2024-04-22T01:14:48.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:48 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:49.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:48 smithi203 ceph-mon[31825]: scrub queued for path: / 2024-04-22T01:14:49.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:48 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:49.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:48 smithi203 ceph-mon[31825]: scrub summary: active paths [/] 2024-04-22T01:14:49.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:48 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:14:49.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:49 smithi111 ceph-mon[31057]: pgmap v429: 129 pgs: 129 active+clean; 1.0 GiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 83 op/s 2024-04-22T01:14:49.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:49 smithi038 ceph-mon[28043]: pgmap v429: 129 pgs: 129 active+clean; 1.0 GiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 83 op/s 2024-04-22T01:14:50.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:49 smithi203 ceph-mon[31825]: pgmap v429: 129 pgs: 129 active+clean; 1.0 GiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 83 op/s 2024-04-22T01:14:51.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:51 smithi111 ceph-mon[31057]: pgmap v430: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-04-22T01:14:51.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:51 smithi111 ceph-mon[31057]: scrub summary: idle 2024-04-22T01:14:51.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:51 smithi038 ceph-mon[28043]: pgmap v430: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-04-22T01:14:51.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:51 smithi038 ceph-mon[28043]: scrub summary: idle 2024-04-22T01:14:52.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:51 smithi203 ceph-mon[31825]: pgmap v430: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-04-22T01:14:52.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:51 smithi203 ceph-mon[31825]: scrub summary: idle 2024-04-22T01:14:53.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:53 smithi111 ceph-mon[31057]: pgmap v431: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 166 MiB/s wr, 82 op/s 2024-04-22T01:14:53.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:53 smithi038 ceph-mon[28043]: pgmap v431: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 166 MiB/s wr, 82 op/s 2024-04-22T01:14:54.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:53 smithi203 ceph-mon[31825]: pgmap v431: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 166 MiB/s wr, 82 op/s 2024-04-22T01:14:55.219 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:55.219 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:55.219 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:55.219 INFO:tasks.workunit.client.0.smithi038.stdout:iozone test complete. 2024-04-22T01:14:55.220 INFO:tasks.workunit.client.0.smithi038.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Iozone: Performance Test of File I/O 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Version $Revision: 3.494 $ 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Compiled for 64 bit mode. 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Build: linux-AMD64 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-22T01:14:55.287 INFO:tasks.workunit.client.0.smithi038.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Run began: Mon Apr 22 01:14:55 2024 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Include close in write timing 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Include fsync in write timing 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: File size set to 10485760 kB 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Record Size 1024 kB 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Output is in kBytes/sec 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Time Resolution = 0.000001 seconds. 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Processor cache size set to 1024 kBytes. 2024-04-22T01:14:55.288 INFO:tasks.workunit.client.0.smithi038.stdout: Processor cache line size set to 32 bytes. 2024-04-22T01:14:55.289 INFO:tasks.workunit.client.0.smithi038.stdout: File stride size set to 17 * record size. 2024-04-22T01:14:55.289 INFO:tasks.workunit.client.0.smithi038.stdout: Throughput test with 1 process 2024-04-22T01:14:55.289 INFO:tasks.workunit.client.0.smithi038.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-04-22T01:14:55.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 -- 172.21.15.38:0/1806230959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 msgr2=0x7f86b4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 --2- 172.21.15.38:0/1806230959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4100ac0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f86a4009900 tx=0x7f86a402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 -- 172.21.15.38:0/1806230959 shutdown_connections 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 --2- 172.21.15.38:0/1806230959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86b4102250 0x7f86b4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 --2- 172.21.15.38:0/1806230959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 0x7f86b4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 --2- 172.21.15.38:0/1806230959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4100ac0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 -- 172.21.15.38:0/1806230959 >> 172.21.15.38:0/1806230959 conn(0x7f86b40fbc10 msgr2=0x7f86b40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 -- 172.21.15.38:0/1806230959 shutdown_connections 2024-04-22T01:14:55.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.312+0000 7f86b8eed640 1 -- 172.21.15.38:0/1806230959 wait complete. 2024-04-22T01:14:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 Processor -- start 2024-04-22T01:14:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 -- start start 2024-04-22T01:14:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 0x7f86b4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86b4102250 0x7f86b419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410c2f0 con 0x7f86b41006a0 2024-04-22T01:14:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410c170 con 0x7f86b4102250 2024-04-22T01:14:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.313+0000 7f86b8eed640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410c470 con 0x7f86b4101890 2024-04-22T01:14:55.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57828/0 (socket says 172.21.15.38:57828) 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 -- 172.21.15.38:0/3150407661 learned_addr learned my addr 172.21.15.38:0/3150407661 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86b4102250 0x7f86b419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 0x7f86b4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 -- 172.21.15.38:0/3150407661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 msgr2=0x7f86b4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 0x7f86b4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 -- 172.21.15.38:0/3150407661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86b4102250 msgr2=0x7f86b419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86b4102250 0x7f86b419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 -- 172.21.15.38:0/3150407661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b419f650 con 0x7f86b41006a0 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 0x7f86b4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b259c640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4197a80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f86a4000c00 tx=0x7f86a4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86a37fe640 1 -- 172.21.15.38:0/3150407661 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a4041070 con 0x7f86b41006a0 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86a37fe640 1 -- 172.21.15.38:0/3150407661 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a40044d0 con 0x7f86b41006a0 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.314+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b419f8e0 con 0x7f86b41006a0 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.315+0000 7f86a37fe640 1 -- 172.21.15.38:0/3150407661 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a4034440 con 0x7f86b41006a0 2024-04-22T01:14:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.315+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b419fdd0 con 0x7f86b41006a0 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.317+0000 7f86a37fe640 1 -- 172.21.15.38:0/3150407661 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f86a4044070 con 0x7f86b41006a0 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 Processor -- start 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 -- start start 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad87150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad8c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad8c890 0x55cf5ad8ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cf5acebb70 con 0x55cf5ad87720 2024-04-22T01:14:55.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cf5ac94470 con 0x55cf5ad86d30 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cf5ace7b30 con 0x55cf5ad8c890 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86b259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad8c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86b259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad8c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57834/0 (socket says 172.21.15.38:57834) 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86b259c640 1 -- 172.21.15.38:0/2032021107 learned_addr learned my addr 172.21.15.38:0/2032021107 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.318+0000 7f86a37fe640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f869c0a2c30 0x7f869c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad8c890 0x55cf5ad8ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad87150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f869c0a2c30 0x7f869c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86a37fe640 1 -- 172.21.15.38:0/3150407661 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f86a40f0fb0 con 0x7f86b41006a0 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b259c640 1 -- 172.21.15.38:0/2032021107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad8c890 msgr2=0x55cf5ad8ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b259c640 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad8c890 0x55cf5ad8ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b259c640 1 -- 172.21.15.38:0/2032021107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 msgr2=0x55cf5ad87150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b259c640 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad87150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b259c640 1 -- 172.21.15.38:0/2032021107 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf5aa00210 con 0x55cf5ad87720 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b259c640 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad8c220 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f86a40ee6e0 tx=0x7f86a402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e722a9da8cf3a7e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f869c0a2c30 0x7f869c0a50f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55cf5ad87f80 tx=0x7f86ac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86a0ff9640 1 -- 172.21.15.38:0/2032021107 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a404b070 con 0x55cf5ad87720 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86a0ff9640 1 -- 172.21.15.38:0/2032021107 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a40bdb40 con 0x55cf5ad87720 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.319+0000 7f86a0ff9640 1 -- 172.21.15.38:0/2032021107 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a40c17f0 con 0x55cf5ad87720 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2032021107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 msgr2=0x55cf5ad8c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad8c220 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f86a40ee6e0 tx=0x7f86a402bc30 comp rx=0 tx=0).stop 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2032021107 shutdown_connections 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad8c890 0x55cf5ad8ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad8c220 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2032021107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad87150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2032021107 >> 172.21.15.38:0/2032021107 conn(0x55cf5ad822a0 msgr2=0x55cf5ad826a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2032021107 shutdown_connections 2024-04-22T01:14:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.320+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2032021107 wait complete. 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 -- start start 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 Processor -- start 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad90d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad91260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad982a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cf5ad8f600 con 0x55cf5ad8c890 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cf5aca7a10 con 0x55cf5ad86d30 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cf5ad8f780 con 0x55cf5ad87720 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86bbbb3740 10 client.0 _ll_register_callbacks cb 0x55cf5ace2b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86b2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86b2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57836/0 (socket says 172.21.15.38:57836) 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3887454910 learned_addr learned my addr 172.21.15.38:0/3887454910 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86b259c640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad90d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.321+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad91260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3887454910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 msgr2=0x55cf5ad91260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad91260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 msgr2=0x55cf5ad90d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad90d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3887454910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf5aacc620 con 0x55cf5ad8c890 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b259c640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad90d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad982a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00fef0 tx=0x7f86ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a75698d7b26edd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f8699ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f8699ffb640 1 -- 172.21.15.38:0/3887454910 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac02e060 con 0x55cf5ad8c890 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f8699ffb640 1 -- 172.21.15.38:0/3887454910 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f86ac021c00 con 0x55cf5ad8c890 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86a40bdb40 con 0x55cf5ad8c890 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f8699ffb640 1 -- 172.21.15.38:0/3887454910 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86ac035050 con 0x55cf5ad8c890 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86bbbb3740 10 client.16257 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf5ad9b680 con 0x55cf5ad8c890 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.322+0000 7f8699ffb640 1 -- 172.21.15.38:0/3887454910 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a40bdb40 con 0x55cf5ad8c890 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f8699ffb640 1 -- 172.21.15.38:0/3887454910 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f86ac01d070 con 0x55cf5ad8c890 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86bbbb3740 10 client.16257 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86bbbb3740 10 client.16257 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86bbbb3740 20 client.16257 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ad9cd70 0x55cf5ad9f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86bbbb3740 4 client.16257 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf5acf90b0 con 0x55cf5ad9cd70 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86b259c640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ad9cd70 0x55cf5ad9f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f86b259c640 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ad9cd70 0x55cf5ad9f200 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85996cd5210cc927 server_cookie=587c2a551d8f1cee in_seq=0 out_seq=0 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.323+0000 7f8699ffb640 10 client.16257 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f8699ffb640 1 -- 172.21.15.38:0/3887454910 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf5acf90b0 con 0x55cf5ad9cd70 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f8699ffb640 10 client.16257 handle_command_reply: tid=0 2024-04-22T01:14:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 client.16257 shutdown 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 msgr2=0x55cf5ad982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad982a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00fef0 tx=0x7f86ac004a40 comp rx=0 tx=0).stop 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 shutdown_connections 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ad9cd70 0x55cf5ad9f200 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad982a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad91260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3887454910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad90d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f8699ffb640 0 client.16257 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 >> 172.21.15.38:0/3887454910 conn(0x55cf5ad822a0 msgr2=0x55cf5ad814c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 shutdown_connections 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.324+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3887454910 wait complete. 2024-04-22T01:14:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.325+0000 7f86bbbb3740 20 client.16257 trim_cache size 0 max 16384 2024-04-22T01:14:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 Processor -- start 2024-04-22T01:14:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 -- start start 2024-04-22T01:14:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad889e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad88f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.340 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad98260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41aa4b0 con 0x55cf5ad8c890 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41abff0 con 0x55cf5ad86d30 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41aab80 con 0x55cf5ad87720 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86b2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad98260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86b1d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad88f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86b2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad98260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57838/0 (socket says 172.21.15.38:57838) 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86b259c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad889e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.340+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3697859538 learned_addr learned my addr 172.21.15.38:0/3697859538 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3697859538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 msgr2=0x55cf5ad88f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad88f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3697859538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 msgr2=0x55cf5ad889e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad889e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b2d9d640 1 -- 172.21.15.38:0/3697859538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf5ad8bc80 con 0x55cf5ad8c890 2024-04-22T01:14:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad88f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:14:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad98260 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f86ac006010 tx=0x7f86ac011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e580a8e7ada306f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f869bfff640 1 -- 172.21.15.38:0/3697859538 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac02e040 con 0x55cf5ad8c890 2024-04-22T01:14:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f869bfff640 1 -- 172.21.15.38:0/3697859538 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ac009ce0 con 0x55cf5ad8c890 2024-04-22T01:14:55.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.341+0000 7f869bfff640 1 -- 172.21.15.38:0/3697859538 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac004030 con 0x55cf5ad8c890 2024-04-22T01:14:55.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3697859538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 msgr2=0x55cf5ad98260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad98260 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f86ac006010 tx=0x7f86ac011b10 comp rx=0 tx=0).stop 2024-04-22T01:14:55.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3697859538 shutdown_connections 2024-04-22T01:14:55.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad8c890 0x55cf5ad98260 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad87720 0x55cf5ad88f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/3697859538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad86d30 0x55cf5ad889e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3697859538 >> 172.21.15.38:0/3697859538 conn(0x55cf5ad822a0 msgr2=0x55cf5ad82680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3697859538 shutdown_connections 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 -- 172.21.15.38:0/3697859538 wait complete. 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 -- start start 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.342+0000 7f86bbbb3740 1 Processor -- start 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad86d30 0x55cf5ad956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad95c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad8c890 0x55cf5ad96c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41ab920 con 0x55cf5ad87720 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41ab250 con 0x55cf5ad8c890 2024-04-22T01:14:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41a8470 con 0x55cf5ad86d30 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86bbbb3740 10 client.0 _ll_register_callbacks cb 0x7f86ac00fd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad95c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad95c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57848/0 (socket says 172.21.15.38:57848) 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 -- 172.21.15.38:0/2906038147 learned_addr learned my addr 172.21.15.38:0/2906038147 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b259c640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad86d30 0x55cf5ad956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad8c890 0x55cf5ad96c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 -- 172.21.15.38:0/2906038147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad86d30 msgr2=0x55cf5ad956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad86d30 0x55cf5ad956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 -- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad8c890 msgr2=0x55cf5ad96c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad8c890 0x55cf5ad96c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 -- 172.21.15.38:0/2906038147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf5ad97200 con 0x55cf5ad87720 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b2d9d640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad8c890 0x55cf5ad96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:14:55.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f86b1d9b640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad95c00 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f86a800bb80 tx=0x7f86a80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2948340d6182a20c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.343+0000 7f8699ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f8699ffb640 1 -- 172.21.15.38:0/2906038147 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8005220 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f8699ffb640 1 -- 172.21.15.38:0/2906038147 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f86a80053c0 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86ac009ce0 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f8699ffb640 1 -- 172.21.15.38:0/2906038147 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86a8027050 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 10 client.16263 fetch_fsmap learned FSMap version 28 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf5ad98610 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f8699ffb640 1 -- 172.21.15.38:0/2906038147 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ac009ce0 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f8699ffb640 1 -- 172.21.15.38:0/2906038147 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f86a801b070 con 0x55cf5ad87720 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 10 client.16263 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 20 client.16263 populate_metadata read hostname 'smithi038' 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ae03bf0 0x55cf5ae06010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 4 client.16263 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:14:55.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.344+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf5acf3400 con 0x55cf5ae03bf0 2024-04-22T01:14:55.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.345+0000 7f86b259c640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ae03bf0 0x55cf5ae06010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:14:55.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.345+0000 7f8699ffb640 10 client.16263 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:55.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.345+0000 7f86b259c640 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ae03bf0 0x55cf5ae06010 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c965a3de3a3f573e server_cookie=54bf707f75e920e2 in_seq=0 out_seq=0 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f8699ffb640 1 -- 172.21.15.38:0/2906038147 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf5acf3400 con 0x55cf5ae03bf0 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f8699ffb640 10 client.16263 handle_command_reply: tid=0 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 client.16263 shutdown 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 msgr2=0x55cf5ad95c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad95c00 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f86a800bb80 tx=0x7f86a80039c0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 shutdown_connections 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55cf5ae03bf0 0x55cf5ae06010 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cf5ad8c890 0x55cf5ad96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cf5ad87720 0x55cf5ad95c00 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f8699ffb640 0 client.16263 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:14:55.359 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 --2- 172.21.15.38:0/2906038147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cf5ad86d30 0x55cf5ad956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 >> 172.21.15.38:0/2906038147 conn(0x55cf5ad822a0 msgr2=0x55cf5ad889e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.359+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 shutdown_connections 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86bbbb3740 1 -- 172.21.15.38:0/2906038147 wait complete. 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86bbbb3740 20 client.16263 trim_cache size 0 max 16384 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:14:55.360 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f869c0a2c30 msgr2=0x7f869c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f869c0a2c30 0x7f869c0a50f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55cf5ad87f80 tx=0x7f86ac009500 comp rx=0 tx=0).stop 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 msgr2=0x7f86b4197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4197a80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f86a4000c00 tx=0x7f86a4005d50 comp rx=0 tx=0).stop 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 shutdown_connections 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f869c0a2c30 0x7f869c0a50f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86b4102250 0x7f86b419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b4101890 0x7f86b4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 --2- 172.21.15.38:0/3150407661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86b41006a0 0x7f86b4197a80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:14:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 >> 172.21.15.38:0/3150407661 conn(0x7f86b40fbc10 msgr2=0x7f86b40fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:14:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 shutdown_connections 2024-04-22T01:14:55.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:14:55.360+0000 7f86b8eed640 1 -- 172.21.15.38:0/3150407661 wait complete. 2024-04-22T01:14:55.896 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:55 smithi111 ceph-mon[31057]: pgmap v432: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 119 MiB/s wr, 66 op/s 2024-04-22T01:14:55.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:55 smithi038 ceph-mon[28043]: pgmap v432: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 119 MiB/s wr, 66 op/s 2024-04-22T01:14:56.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:55 smithi203 ceph-mon[31825]: pgmap v432: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 119 MiB/s wr, 66 op/s 2024-04-22T01:14:57.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:57 smithi111 ceph-mon[31057]: pgmap v433: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 97 MiB/s wr, 59 op/s 2024-04-22T01:14:57.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:57 smithi038 ceph-mon[28043]: pgmap v433: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 97 MiB/s wr, 59 op/s 2024-04-22T01:14:58.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:57 smithi203 ceph-mon[31825]: pgmap v433: 129 pgs: 129 active+clean; 1.0 GiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 97 MiB/s wr, 59 op/s 2024-04-22T01:14:59.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:14:59 smithi111 ceph-mon[31057]: pgmap v434: 129 pgs: 129 active+clean; 1.3 GiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 67 MiB/s wr, 68 op/s 2024-04-22T01:14:59.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:14:59 smithi038 ceph-mon[28043]: pgmap v434: 129 pgs: 129 active+clean; 1.3 GiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 67 MiB/s wr, 68 op/s 2024-04-22T01:15:00.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:14:59 smithi203 ceph-mon[31825]: pgmap v434: 129 pgs: 129 active+clean; 1.3 GiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 67 MiB/s wr, 68 op/s 2024-04-22T01:15:01.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:01 smithi111 ceph-mon[31057]: pgmap v435: 129 pgs: 129 active+clean; 1.9 GiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 86 MiB/s wr, 59 op/s 2024-04-22T01:15:01.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:01 smithi038 ceph-mon[28043]: pgmap v435: 129 pgs: 129 active+clean; 1.9 GiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 86 MiB/s wr, 59 op/s 2024-04-22T01:15:02.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:01 smithi203 ceph-mon[31825]: pgmap v435: 129 pgs: 129 active+clean; 1.9 GiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 86 MiB/s wr, 59 op/s 2024-04-22T01:15:03.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:03 smithi111 ceph-mon[31057]: pgmap v436: 129 pgs: 129 active+clean; 3.8 GiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 291 MiB/s wr, 260 op/s 2024-04-22T01:15:03.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:03 smithi038 ceph-mon[28043]: pgmap v436: 129 pgs: 129 active+clean; 3.8 GiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 291 MiB/s wr, 260 op/s 2024-04-22T01:15:04.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:03 smithi203 ceph-mon[31825]: pgmap v436: 129 pgs: 129 active+clean; 3.8 GiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 291 MiB/s wr, 260 op/s 2024-04-22T01:15:05.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:05 smithi111 ceph-mon[31057]: pgmap v437: 129 pgs: 129 active+clean; 4.6 GiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 377 MiB/s wr, 283 op/s 2024-04-22T01:15:05.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:05 smithi038 ceph-mon[28043]: pgmap v437: 129 pgs: 129 active+clean; 4.6 GiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 377 MiB/s wr, 283 op/s 2024-04-22T01:15:06.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:05 smithi203 ceph-mon[31825]: pgmap v437: 129 pgs: 129 active+clean; 4.6 GiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 377 MiB/s wr, 283 op/s 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 initial writers = 923766.62 kB/sec 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 initial writers = 923748.17 kB/sec 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 923766.62 kB/sec 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 923766.62 kB/sec 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 923766.62 kB/sec 2024-04-22T01:15:06.674 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 10485760.00 kB 2024-04-22T01:15:07.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:07 smithi111 ceph-mon[31057]: pgmap v438: 129 pgs: 129 active+clean; 6.5 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 557 MiB/s wr, 387 op/s 2024-04-22T01:15:07.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:07 smithi038 ceph-mon[28043]: pgmap v438: 129 pgs: 129 active+clean; 6.5 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 557 MiB/s wr, 387 op/s 2024-04-22T01:15:08.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:07 smithi203 ceph-mon[31825]: pgmap v438: 129 pgs: 129 active+clean; 6.5 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 557 MiB/s wr, 387 op/s 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 -- 172.21.15.38:0/263937348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28001006d0 msgr2=0x7f2800100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 --2- 172.21.15.38:0/263937348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28001006d0 0x7f2800100af0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009900 tx=0x7f27fc02bc50 comp rx=0 tx=0).stop 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 -- 172.21.15.38:0/263937348 shutdown_connections 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 --2- 172.21.15.38:0/263937348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2800102280 0x7f2800108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 --2- 172.21.15.38:0/263937348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28001018c0 0x7f2800101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 --2- 172.21.15.38:0/263937348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28001006d0 0x7f2800100af0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 -- 172.21.15.38:0/263937348 >> 172.21.15.38:0/263937348 conn(0x7f28000fbc40 msgr2=0x7f28000fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 -- 172.21.15.38:0/263937348 shutdown_connections 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.462+0000 7f280858a640 1 -- 172.21.15.38:0/263937348 wait complete. 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 Processor -- start 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 -- start start 2024-04-22T01:15:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28001006d0 0x7f2800104660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28001018c0 0x7f2800102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 0x7f28001031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f280010c320 con 0x7f2800102280 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f280010c1a0 con 0x7f28001006d0 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f280858a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f280010c4a0 con 0x7f28001018c0 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f2806b27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 0x7f28001031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.463+0000 7f2806b27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 0x7f28001031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37134/0 (socket says 172.21.15.38:37134) 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 -- 172.21.15.38:0/4044602310 learned_addr learned my addr 172.21.15.38:0/4044602310 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2805b25640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28001018c0 0x7f2800102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806326640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28001006d0 0x7f2800104660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 -- 172.21.15.38:0/4044602310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28001018c0 msgr2=0x7f2800102ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28001018c0 0x7f2800102ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 -- 172.21.15.38:0/4044602310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28001006d0 msgr2=0x7f2800104660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28001006d0 0x7f2800104660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 -- 172.21.15.38:0/4044602310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2800103a70 con 0x7f2800102280 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806326640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28001006d0 0x7f2800104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f2806b27640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 0x7f28001031e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009fd0 tx=0x7f27f800def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f27f77fe640 1 -- 172.21.15.38:0/4044602310 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8017070 con 0x7f2800102280 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f27f77fe640 1 -- 172.21.15.38:0/4044602310 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f8003e50 con 0x7f2800102280 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28001a1be0 con 0x7f2800102280 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f27f77fe640 1 -- 172.21.15.38:0/4044602310 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f80104c0 con 0x7f2800102280 2024-04-22T01:15:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.464+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28001a20e0 con 0x7f2800102280 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.467+0000 7f27f77fe640 1 -- 172.21.15.38:0/4044602310 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f27f8003970 con 0x7f2800102280 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.467+0000 7f280b173740 1 Processor -- start 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 -- start start 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab82fff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83004220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83006d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ab82f63b70 con 0x55ab83004890 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ab82f0c470 con 0x55ab82ffed30 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f280b173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab82f5fb30 con 0x55ab82fff720 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83006d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83006d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37144/0 (socket says 172.21.15.38:37144) 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 -- 172.21.15.38:0/3373887025 learned_addr learned my addr 172.21.15.38:0/3373887025 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2806326640 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83004220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f27f77fe640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27ec0a2c30 0x7f27ec0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab82fff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 -- 172.21.15.38:0/3373887025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 msgr2=0x55ab83004220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83004220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 -- 172.21.15.38:0/3373887025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 msgr2=0x55ab82fff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab82fff150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f2805b25640 1 -- 172.21.15.38:0/3373887025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab82c78210 con 0x55ab83004890 2024-04-22T01:15:08.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.468+0000 7f27f77fe640 1 -- 172.21.15.38:0/4044602310 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f27f80cf140 con 0x7f2800102280 2024-04-22T01:15:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.469+0000 7f2806b27640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27ec0a2c30 0x7f27ec0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.469+0000 7f2805b25640 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83006d20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f27f000a5c0 tx=0x7f27f000a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0057295ab3bb6f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.469+0000 7f27f4bfe640 1 -- 172.21.15.38:0/3373887025 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f000f910 con 0x55ab83004890 2024-04-22T01:15:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.469+0000 7f27f4bfe640 1 -- 172.21.15.38:0/3373887025 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f001b390 con 0x55ab83004890 2024-04-22T01:15:08.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.469+0000 7f27f4bfe640 1 -- 172.21.15.38:0/3373887025 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f000d540 con 0x55ab83004890 2024-04-22T01:15:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 -- 172.21.15.38:0/3373887025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 msgr2=0x55ab83006d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f2806b27640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27ec0a2c30 0x7f27ec0a50f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f27f8097a30 tx=0x7f27f8004020 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83006d20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f27f000a5c0 tx=0x7f27f000a810 comp rx=0 tx=0).stop 2024-04-22T01:15:08.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 -- 172.21.15.38:0/3373887025 shutdown_connections 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83006d20 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83004220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 --2- 172.21.15.38:0/3373887025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab82fff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 -- 172.21.15.38:0/3373887025 >> 172.21.15.38:0/3373887025 conn(0x55ab82ffa2a0 msgr2=0x55ab82ffa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 -- 172.21.15.38:0/3373887025 shutdown_connections 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 -- 172.21.15.38:0/3373887025 wait complete. 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 -- start start 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.470+0000 7f280b173740 1 Processor -- start 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82ffed30 0x55ab83009040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab83009580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab83004890 0x55ab830105c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ab83007840 con 0x55ab82fff720 2024-04-22T01:15:08.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ab82f1fa10 con 0x55ab83004890 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab830079c0 con 0x55ab82ffed30 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 10 client.0 _ll_register_callbacks cb 0x55ab82f5ab60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab83009580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab83009580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37154/0 (socket says 172.21.15.38:37154) 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 -- 172.21.15.38:0/1186153664 learned_addr learned my addr 172.21.15.38:0/1186153664 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2805b25640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82ffed30 0x55ab83009040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2805b25640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab83004890 0x55ab830105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 -- 172.21.15.38:0/1186153664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82ffed30 msgr2=0x55ab83009040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82ffed30 0x55ab83009040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 -- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab83004890 msgr2=0x55ab830105c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab83004890 0x55ab830105c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 -- 172.21.15.38:0/1186153664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab82d44620 con 0x55ab82fff720 2024-04-22T01:15:08.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2805b25640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab83004890 0x55ab830105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f2806326640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab83009580 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009900 tx=0x7f27fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da50de027d15b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f27e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f27e9ffb640 1 -- 172.21.15.38:0/1186153664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc005230 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f27e9ffb640 1 -- 172.21.15.38:0/1186153664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f27fc0053d0 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.471+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27f001b390 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f27e9ffb640 1 -- 172.21.15.38:0/1186153664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27fc04e050 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 10 client.16275 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab830139a0 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f27e9ffb640 1 -- 172.21.15.38:0/1186153664 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f001b390 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f27e9ffb640 1 -- 172.21.15.38:0/1186153664 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f27fc0038c0 con 0x55ab82fff720 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 10 client.16275 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 20 client.16275 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:08.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab83015130 0x55ab83017560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 4 client.16275 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab82f710b0 con 0x55ab83015130 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.472+0000 7f2805b25640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab83015130 0x55ab83017560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.473+0000 7f2805b25640 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab83015130 0x55ab83017560 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e530254a6303629d server_cookie=727910478a5508d5 in_seq=0 out_seq=0 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.473+0000 7f27e9ffb640 10 client.16275 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f27e9ffb640 1 -- 172.21.15.38:0/1186153664 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab82f710b0 con 0x55ab83015130 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f27e9ffb640 10 client.16275 handle_command_reply: tid=0 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 client.16275 shutdown 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 msgr2=0x55ab83009580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab83009580 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009900 tx=0x7f27fc0040c0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 shutdown_connections 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab83015130 0x55ab83017560 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab83004890 0x55ab830105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab83009580 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f27e9ffb640 0 client.16275 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 --2- 172.21.15.38:0/1186153664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82ffed30 0x55ab83009040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 >> 172.21.15.38:0/1186153664 conn(0x55ab82ffa2a0 msgr2=0x55ab82ff94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 shutdown_connections 2024-04-22T01:15:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 1 -- 172.21.15.38:0/1186153664 wait complete. 2024-04-22T01:15:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.474+0000 7f280b173740 20 client.16275 trim_cache size 0 max 16384 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 Processor -- start 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 -- start start 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab83000ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83000ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83010580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28001ac710 con 0x55ab83004890 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f28001ae250 con 0x55ab82ffed30 2024-04-22T01:15:08.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f280b173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f28001acde0 con 0x55ab82fff720 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83010580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83010580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37162/0 (socket says 172.21.15.38:37162) 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 -- 172.21.15.38:0/1892071987 learned_addr learned my addr 172.21.15.38:0/1892071987 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab83000ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2806326640 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83000ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 -- 172.21.15.38:0/1892071987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 msgr2=0x55ab83000ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83000ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 -- 172.21.15.38:0/1892071987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 msgr2=0x55ab83000ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab83000ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 -- 172.21.15.38:0/1892071987 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab82fffb00 con 0x55ab83004890 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2806326640 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83000ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.490+0000 7f2805b25640 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83010580 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f27f0005890 tx=0x7f27f001e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116ea7432993f5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f27ebfff640 1 -- 172.21.15.38:0/1892071987 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0021040 con 0x55ab83004890 2024-04-22T01:15:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f27ebfff640 1 -- 172.21.15.38:0/1892071987 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f001b390 con 0x55ab83004890 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f27ebfff640 1 -- 172.21.15.38:0/1892071987 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f001b900 con 0x55ab83004890 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 -- 172.21.15.38:0/1892071987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 msgr2=0x55ab83010580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83010580 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f27f0005890 tx=0x7f27f001e040 comp rx=0 tx=0).stop 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 -- 172.21.15.38:0/1892071987 shutdown_connections 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab83004890 0x55ab83010580 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab82fff720 0x55ab83000ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 --2- 172.21.15.38:0/1892071987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab83000ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 -- 172.21.15.38:0/1892071987 >> 172.21.15.38:0/1892071987 conn(0x55ab82ffa2a0 msgr2=0x55ab82ffa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 -- 172.21.15.38:0/1892071987 shutdown_connections 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.491+0000 7f280b173740 1 -- 172.21.15.38:0/1892071987 wait complete. 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 -- start start 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 Processor -- start 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab8300d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab8300dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab83004890 0x55ab8300eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28001adb80 con 0x55ab82fff720 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f28001ad4b0 con 0x55ab82ffed30 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f28001aa6d0 con 0x55ab83004890 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f280b173740 10 client.0 _ll_register_callbacks cb 0x7f27fc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab8300dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab8300dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37168/0 (socket says 172.21.15.38:37168) 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 -- 172.21.15.38:0/2285125471 learned_addr learned my addr 172.21.15.38:0/2285125471 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2805b25640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab83004890 0x55ab8300eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2805b25640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab8300d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 -- 172.21.15.38:0/2285125471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab83004890 msgr2=0x55ab8300eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab83004890 0x55ab8300eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 -- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 msgr2=0x55ab8300d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab8300d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 -- 172.21.15.38:0/2285125471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab8300f290 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2805b25640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab8300d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f2806326640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab8300dc90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009850 tx=0x7f27fc005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44c54628d428849 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.492+0000 7f27e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f27e9ffb640 1 -- 172.21.15.38:0/2285125471 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0037f0 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f27e9ffb640 1 -- 172.21.15.38:0/2285125471 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f27fc003990 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27f001b390 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f27e9ffb640 1 -- 172.21.15.38:0/2285125471 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27fc060050 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 10 client.16281 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab83010610 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f27e9ffb640 1 -- 172.21.15.38:0/2285125471 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f001b390 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f27e9ffb640 1 -- 172.21.15.38:0/2285125471 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f27fc034ab0 con 0x55ab82fff720 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 10 client.16281 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 10 client.16281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 20 client.16281 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab8307bc60 0x55ab8307e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:08.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 4 client.16281 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:15:08.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.493+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab82f6b400 con 0x55ab8307bc60 2024-04-22T01:15:08.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.494+0000 7f2805b25640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab8307bc60 0x55ab8307e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:08.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.494+0000 7f2805b25640 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab8307bc60 0x55ab8307e080 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d99a3670d074060 server_cookie=ef45f2a9f568eab6 in_seq=0 out_seq=0 2024-04-22T01:15:08.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.494+0000 7f27e9ffb640 10 client.16281 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f27e9ffb640 1 -- 172.21.15.38:0/2285125471 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab82f6b400 con 0x55ab8307bc60 2024-04-22T01:15:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f27e9ffb640 10 client.16281 handle_command_reply: tid=0 2024-04-22T01:15:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 client.16281 shutdown 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 msgr2=0x55ab8300dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab8300dc90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009850 tx=0x7f27fc005870 comp rx=0 tx=0).stop 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 shutdown_connections 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55ab8307bc60 0x55ab8307e080 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab83004890 0x55ab8300eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ab82fff720 0x55ab8300dc90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f27e9ffb640 0 client.16281 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 --2- 172.21.15.38:0/2285125471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab82ffed30 0x55ab8300d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 >> 172.21.15.38:0/2285125471 conn(0x55ab82ffa2a0 msgr2=0x55ab83000ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 shutdown_connections 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 1 -- 172.21.15.38:0/2285125471 wait complete. 2024-04-22T01:15:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.508+0000 7f280b173740 20 client.16281 trim_cache size 0 max 16384 2024-04-22T01:15:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:15:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:15:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:15:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27ec0a2c30 msgr2=0x7f27ec0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27ec0a2c30 0x7f27ec0a50f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f27f8097a30 tx=0x7f27f8004020 comp rx=0 tx=0).stop 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 msgr2=0x7f28001031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 0x7f28001031e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009fd0 tx=0x7f27f800def0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 shutdown_connections 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f27ec0a2c30 0x7f27ec0a50f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2800102280 0x7f28001031e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28001018c0 0x7f2800102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 --2- 172.21.15.38:0/4044602310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28001006d0 0x7f2800104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 >> 172.21.15.38:0/4044602310 conn(0x7f28000fbc40 msgr2=0x7f28000fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 shutdown_connections 2024-04-22T01:15:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:08.509+0000 7f280858a640 1 -- 172.21.15.38:0/4044602310 wait complete. 2024-04-22T01:15:09.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:09 smithi111 ceph-mon[31057]: pgmap v439: 129 pgs: 129 active+clean; 9.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 769 MiB/s wr, 442 op/s 2024-04-22T01:15:09.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:09 smithi038 ceph-mon[28043]: pgmap v439: 129 pgs: 129 active+clean; 9.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 769 MiB/s wr, 442 op/s 2024-04-22T01:15:10.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:09 smithi203 ceph-mon[31825]: pgmap v439: 129 pgs: 129 active+clean; 9.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 769 MiB/s wr, 442 op/s 2024-04-22T01:15:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:11 smithi111 ceph-mon[31057]: pgmap v440: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 819 MiB/s wr, 429 op/s 2024-04-22T01:15:11.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:11 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:15:11.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:11 smithi038 ceph-mon[28043]: pgmap v440: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 819 MiB/s wr, 429 op/s 2024-04-22T01:15:11.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:11 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:15:12.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:11 smithi203 ceph-mon[31825]: pgmap v440: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 819 MiB/s wr, 429 op/s 2024-04-22T01:15:12.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:11 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:15:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:13 smithi111 ceph-mon[31057]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 801 MiB/s wr, 424 op/s 2024-04-22T01:15:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:15:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:15:13.895 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:15:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:13 smithi038 ceph-mon[28043]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 801 MiB/s wr, 424 op/s 2024-04-22T01:15:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:15:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:15:13.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:15:14.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:13 smithi203 ceph-mon[31825]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 801 MiB/s wr, 424 op/s 2024-04-22T01:15:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:15:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:15:14.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:15:15.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:15 smithi038 ceph-mon[28043]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 642 MiB/s wr, 237 op/s 2024-04-22T01:15:16.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:15 smithi203 ceph-mon[31825]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 642 MiB/s wr, 237 op/s 2024-04-22T01:15:16.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:15 smithi111 ceph-mon[31057]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 642 MiB/s wr, 237 op/s 2024-04-22T01:15:17.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:17 smithi038 ceph-mon[28043]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 683 MiB/s wr, 340 op/s 2024-04-22T01:15:18.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:17 smithi203 ceph-mon[31825]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 683 MiB/s wr, 340 op/s 2024-04-22T01:15:18.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:17 smithi111 ceph-mon[31057]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 683 MiB/s wr, 340 op/s 2024-04-22T01:15:18.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:15:18.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:15:18.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.884+0000 7f577239f640 1 -- 172.21.15.38:0/2795500801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 msgr2=0x7f57640ad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.884+0000 7f577239f640 1 --2- 172.21.15.38:0/2795500801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57640ad420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5768009860 tx=0x7f576802bbb0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.889+0000 7f577239f640 1 -- 172.21.15.38:0/2795500801 shutdown_connections 2024-04-22T01:15:18.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.889+0000 7f577239f640 1 --2- 172.21.15.38:0/2795500801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57640ad420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.889+0000 7f577239f640 1 --2- 172.21.15.38:0/2795500801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57640a61d0 0x7f57640a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.889+0000 7f577239f640 1 --2- 172.21.15.38:0/2795500801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 0x7f57640a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.889+0000 7f577239f640 1 -- 172.21.15.38:0/2795500801 >> 172.21.15.38:0/2795500801 conn(0x7f57640a0530 msgr2=0x7f57640a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.894+0000 7f577239f640 1 -- 172.21.15.38:0/2795500801 shutdown_connections 2024-04-22T01:15:18.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.898+0000 7f577239f640 1 -- 172.21.15.38:0/2795500801 wait complete. 2024-04-22T01:15:18.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.899+0000 7f577239f640 1 Processor -- start 2024-04-22T01:15:18.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.900+0000 7f577239f640 1 -- start start 2024-04-22T01:15:18.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.901+0000 7f577239f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 0x7f576413af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.901+0000 7f577239f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57640a61d0 0x7f576413b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.901+0000 7f577239f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57641424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.901+0000 7f577239f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f57640b07f0 con 0x7f57640a4fe0 2024-04-22T01:15:18.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.901+0000 7f577239f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f57640b0670 con 0x7f57640a61d0 2024-04-22T01:15:18.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.901+0000 7f577239f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f57640b0970 con 0x7f57640a6b90 2024-04-22T01:15:18.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.904+0000 7f5771b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57641424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.905+0000 7f5771b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57641424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:41976/0 (socket says 172.21.15.38:41976) 2024-04-22T01:15:18.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.905+0000 7f5771b9e640 1 -- 172.21.15.38:0/3164392114 learned_addr learned my addr 172.21.15.38:0/3164392114 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:18.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.905+0000 7f5770b9c640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57640a61d0 0x7f576413b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.907+0000 7f577139d640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 0x7f576413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.910+0000 7f577139d640 1 -- 172.21.15.38:0/3164392114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 msgr2=0x7f57641424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.910+0000 7f577139d640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57641424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.910+0000 7f577139d640 1 -- 172.21.15.38:0/3164392114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57640a61d0 msgr2=0x7f576413b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.910+0000 7f577139d640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57640a61d0 0x7f576413b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.910+0000 7f577139d640 1 -- 172.21.15.38:0/3164392114 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5764153b80 con 0x7f57640a4fe0 2024-04-22T01:15:18.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.910+0000 7f577139d640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 0x7f576413af30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f57640a6030 tx=0x7f576c06d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.911+0000 7f57627fc640 1 -- 172.21.15.38:0/3164392114 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c07d070 con 0x7f57640a4fe0 2024-04-22T01:15:18.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.911+0000 7f57627fc640 1 -- 172.21.15.38:0/3164392114 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f576c0783f0 con 0x7f57640a4fe0 2024-04-22T01:15:18.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.911+0000 7f57627fc640 1 -- 172.21.15.38:0/3164392114 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0775b0 con 0x7f57640a4fe0 2024-04-22T01:15:18.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.913+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5764153e70 con 0x7f57640a4fe0 2024-04-22T01:15:18.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.913+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5764154220 con 0x7f57640a4fe0 2024-04-22T01:15:18.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.920+0000 7f57761ee740 1 Processor -- start 2024-04-22T01:15:18.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.922+0000 7f57761ee740 1 -- start start 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.923+0000 7f57761ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3f0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.923+0000 7f57761ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.923+0000 7f57761ee740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.923+0000 7f57761ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56134b355560 con 0x56134b3f0af0 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.923+0000 7f57761ee740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56134b2fd730 con 0x56134b3f5c60 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.924+0000 7f57761ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56134b351520 con 0x56134b3f0100 2024-04-22T01:15:18.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.924+0000 7f57627fc640 1 -- 172.21.15.38:0/3164392114 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f576c0695a0 con 0x7f57640a4fe0 2024-04-22T01:15:18.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.925+0000 7f57627fc640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f57580a2d10 0x7f57580a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.925+0000 7f5771b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.926+0000 7f5770b9c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.926+0000 7f5771b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3f0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:41992/0 (socket says 172.21.15.38:41992) 2024-04-22T01:15:18.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.926+0000 7f5771b9e640 1 -- 172.21.15.38:0/346437384 learned_addr learned my addr 172.21.15.38:0/346437384 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:18.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.926+0000 7f5771b9e640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f57580a2d10 0x7f57580a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.926+0000 7f57627fc640 1 -- 172.21.15.38:0/3164392114 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f576c0fc360 con 0x7f57640a4fe0 2024-04-22T01:15:18.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.927+0000 7f5771b9e640 1 -- 172.21.15.38:0/346437384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 msgr2=0x56134b3f80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.927+0000 7f5771b9e640 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.927+0000 7f5771b9e640 1 -- 172.21.15.38:0/346437384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 msgr2=0x56134b3f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.927+0000 7f5771b9e640 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.927+0000 7f5771b9e640 1 -- 172.21.15.38:0/346437384 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134b1337d0 con 0x56134b3f0100 2024-04-22T01:15:18.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.929+0000 7f5771b9e640 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3f0520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5768035eb0 tx=0x7f5768035ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5ca03becc7d5cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.929+0000 7f5771b9e640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f57580a2d10 0x7f57580a51d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f576803b000 tx=0x7f5768048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.929+0000 7f57577fe640 1 -- 172.21.15.38:0/346437384 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768050070 con 0x56134b3f0100 2024-04-22T01:15:18.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.930+0000 7f57577fe640 1 -- 172.21.15.38:0/346437384 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5768047070 con 0x56134b3f0100 2024-04-22T01:15:18.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 -- 172.21.15.38:0/346437384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 msgr2=0x56134b3f0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3f0520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5768035eb0 tx=0x7f5768035ee0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 -- 172.21.15.38:0/346437384 shutdown_connections 2024-04-22T01:15:18.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 --2- 172.21.15.38:0/346437384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3f0520 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.931+0000 7f57761ee740 1 -- 172.21.15.38:0/346437384 >> 172.21.15.38:0/346437384 conn(0x56134b3eb670 msgr2=0x56134b3eba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.932+0000 7f57761ee740 1 -- 172.21.15.38:0/346437384 shutdown_connections 2024-04-22T01:15:18.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.932+0000 7f57761ee740 1 -- 172.21.15.38:0/346437384 wait complete. 2024-04-22T01:15:18.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.932+0000 7f57761ee740 1 -- start start 2024-04-22T01:15:18.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 Processor -- start 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3fa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3fab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b401ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56134b3f8c50 con 0x56134b3f0af0 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56134b310e00 con 0x56134b3f5c60 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56134b3f8dd0 con 0x56134b3f0100 2024-04-22T01:15:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.933+0000 7f57761ee740 10 client.0 _ll_register_callbacks cb 0x56134b34beb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:18.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.935+0000 7f5771b9e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b401ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.935+0000 7f5771b9e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b401ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55436/0 (socket says 172.21.15.38:55436) 2024-04-22T01:15:18.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.935+0000 7f5771b9e640 1 -- 172.21.15.38:0/2914729204 learned_addr learned my addr 172.21.15.38:0/2914729204 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:18.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.935+0000 7f577139d640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3fa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.936+0000 7f5770b9c640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3fab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.936+0000 7f5771b9e640 1 -- 172.21.15.38:0/2914729204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 msgr2=0x56134b3fa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.936+0000 7f5771b9e640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3fa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.936+0000 7f5771b9e640 1 -- 172.21.15.38:0/2914729204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 msgr2=0x56134b3fab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:18.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.936+0000 7f5771b9e640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3fab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.937+0000 7f5771b9e640 1 -- 172.21.15.38:0/2914729204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56134b0693f0 con 0x56134b3f5c60 2024-04-22T01:15:18.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.937+0000 7f577139d640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3fa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:18.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.937+0000 7f5770b9c640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3fab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:18.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.938+0000 7f5771b9e640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b401ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5768009250 tx=0x7f5768005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc041bb77362be2f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:18.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.938+0000 7f5754ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:15:18.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.939+0000 7f5754ff9640 1 -- 172.21.15.38:0/2914729204 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768050040 con 0x56134b3f5c60 2024-04-22T01:15:18.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.939+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5768047070 con 0x56134b3f5c60 2024-04-22T01:15:18.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.939+0000 7f5754ff9640 1 -- 172.21.15.38:0/2914729204 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f5768034d60 con 0x56134b3f5c60 2024-04-22T01:15:18.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.940+0000 7f5754ff9640 1 -- 172.21.15.38:0/2914729204 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5768061050 con 0x56134b3f5c60 2024-04-22T01:15:18.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.940+0000 7f57761ee740 10 client.25790 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:18.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.940+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56134b0290f0 con 0x56134b3f5c60 2024-04-22T01:15:18.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.944+0000 7f5754ff9640 1 -- 172.21.15.38:0/2914729204 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5768047070 con 0x56134b3f5c60 2024-04-22T01:15:18.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.944+0000 7f5754ff9640 1 -- 172.21.15.38:0/2914729204 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f576805c030 con 0x56134b3f5c60 2024-04-22T01:15:18.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.952+0000 7f57761ee740 10 client.25790 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:18.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.952+0000 7f57761ee740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:18.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.952+0000 7f57761ee740 20 client.25790 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:18.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.952+0000 7f57761ee740 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b406090 0x56134b408480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:18.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.952+0000 7f57761ee740 4 client.25790 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:18.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.952+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56134b362aa0 con 0x56134b406090 2024-04-22T01:15:18.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.956+0000 7f577139d640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b406090 0x56134b408480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:18.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.965+0000 7f577139d640 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b406090 0x56134b408480 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a17c4b0b485ca95 server_cookie=5198efe258f79945 in_seq=0 out_seq=0 2024-04-22T01:15:18.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.966+0000 7f5754ff9640 10 client.25790 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:18.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.970+0000 7f5754ff9640 1 -- 172.21.15.38:0/2914729204 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f576c10dae0 con 0x56134b406090 2024-04-22T01:15:18.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.970+0000 7f5754ff9640 10 client.25790 handle_command_reply: tid=0 2024-04-22T01:15:18.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.971+0000 7f57761ee740 1 client.25790 shutdown 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.978+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 msgr2=0x56134b401ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.978+0000 7f57761ee740 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b401ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5768009250 tx=0x7f5768005f10 comp rx=0 tx=0).stop 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 shutdown_connections 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b406090 0x56134b408480 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b401ba0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0af0 0x56134b3fab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 --2- 172.21.15.38:0/2914729204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0100 0x56134b3fa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:18.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 >> 172.21.15.38:0/2914729204 conn(0x56134b3eb670 msgr2=0x56134b3ea800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:18.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f5754ff9640 0 client.25790 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:18.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 shutdown_connections 2024-04-22T01:15:18.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.979+0000 7f57761ee740 1 -- 172.21.15.38:0/2914729204 wait complete. 2024-04-22T01:15:18.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:18.986+0000 7f57761ee740 20 client.25790 trim_cache size 0 max 16384 2024-04-22T01:15:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.032+0000 7f57761ee740 1 Processor -- start 2024-04-22T01:15:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.033+0000 7f57761ee740 1 -- start start 2024-04-22T01:15:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.033+0000 7f57761ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0100 0x56134b3eba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.033+0000 7f57761ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b401b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.034+0000 7f57761ee740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.034+0000 7f57761ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f576415ea70 con 0x56134b3f0100 2024-04-22T01:15:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.034+0000 7f57761ee740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f57641605b0 con 0x56134b3f5c60 2024-04-22T01:15:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.034+0000 7f57761ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f576415f140 con 0x56134b3f0af0 2024-04-22T01:15:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.035+0000 7f5770b9c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b401b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.035+0000 7f5770b9c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b401b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42010/0 (socket says 172.21.15.38:42010) 2024-04-22T01:15:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.035+0000 7f5770b9c640 1 -- 172.21.15.38:0/2793636742 learned_addr learned my addr 172.21.15.38:0/2793636742 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.035+0000 7f577139d640 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0100 0x56134b3eba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.036+0000 7f5771b9e640 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.039+0000 7f577139d640 1 -- 172.21.15.38:0/2793636742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 msgr2=0x56134b401b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.039+0000 7f577139d640 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b401b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.040+0000 7f577139d640 1 -- 172.21.15.38:0/2793636742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 msgr2=0x56134b3f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.040+0000 7f577139d640 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.040+0000 7f577139d640 1 -- 172.21.15.38:0/2793636742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56134b3e70d0 con 0x56134b3f0100 2024-04-22T01:15:19.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.041+0000 7f577139d640 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0100 0x56134b3eba50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f576c0f8d90 tx=0x7f576c11e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d277923807e574 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.048+0000 7f5756ffd640 1 -- 172.21.15.38:0/2793636742 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0facf0 con 0x56134b3f0100 2024-04-22T01:15:19.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.048+0000 7f5756ffd640 1 -- 172.21.15.38:0/2793636742 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56134b362aa0 con 0x56134b3f0100 2024-04-22T01:15:19.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.048+0000 7f5756ffd640 1 -- 172.21.15.38:0/2793636742 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0fae90 con 0x56134b3f0100 2024-04-22T01:15:19.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.051+0000 7f57761ee740 1 -- 172.21.15.38:0/2793636742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0100 msgr2=0x56134b3eba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.051+0000 7f57761ee740 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0100 0x56134b3eba50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f576c0f8d90 tx=0x7f576c11e040 comp rx=0 tx=0).stop 2024-04-22T01:15:19.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.055+0000 7f57761ee740 1 -- 172.21.15.38:0/2793636742 shutdown_connections 2024-04-22T01:15:19.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.055+0000 7f57761ee740 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.055+0000 7f57761ee740 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b401b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.055+0000 7f57761ee740 1 --2- 172.21.15.38:0/2793636742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b3f0100 0x56134b3eba50 secure :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f576c0f8d90 tx=0x7f576c11e040 comp rx=0 tx=0).stop 2024-04-22T01:15:19.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.055+0000 7f57761ee740 1 -- 172.21.15.38:0/2793636742 >> 172.21.15.38:0/2793636742 conn(0x56134b3eb670 msgr2=0x56134b3fa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.059+0000 7f57761ee740 1 -- 172.21.15.38:0/2793636742 shutdown_connections 2024-04-22T01:15:19.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.065+0000 7f57761ee740 1 -- 172.21.15.38:0/2793636742 wait complete. 2024-04-22T01:15:19.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.066+0000 7f57761ee740 1 -- start start 2024-04-22T01:15:19.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.066+0000 7f57761ee740 1 Processor -- start 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b3ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3ff610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b400620 0x56134b400aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f576415fee0 con 0x56134b400620 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f576415f810 con 0x56134b3f5c60 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f576415ca30 con 0x56134b3f0af0 2024-04-22T01:15:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.072+0000 7f57761ee740 10 client.0 _ll_register_callbacks cb 0x7f576804e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.076+0000 7f5770b9c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.077+0000 7f577139d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b3ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.077+0000 7f5770b9c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3ff610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55458/0 (socket says 172.21.15.38:55458) 2024-04-22T01:15:19.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.077+0000 7f5770b9c640 1 -- 172.21.15.38:0/2626661059 learned_addr learned my addr 172.21.15.38:0/2626661059 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:19.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.080+0000 7f5771b9e640 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b400620 0x56134b400aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.082+0000 7f5770b9c640 1 -- 172.21.15.38:0/2626661059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 msgr2=0x56134b3ff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.082+0000 7f5770b9c640 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b3ff0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.082+0000 7f5770b9c640 1 -- 172.21.15.38:0/2626661059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b400620 msgr2=0x56134b400aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:19.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.082+0000 7f5770b9c640 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b400620 0x56134b400aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.082+0000 7f5770b9c640 1 -- 172.21.15.38:0/2626661059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56134b400fe0 con 0x56134b3f5c60 2024-04-22T01:15:19.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.091+0000 7f5770b9c640 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3ff610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f575c010010 tx=0x7f575c0138f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b3b93c1dbd66042 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.091+0000 7f5754ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.094+0000 7f5754ff9640 1 -- 172.21.15.38:0/2626661059 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c021060 con 0x56134b3f5c60 2024-04-22T01:15:19.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.094+0000 7f5754ff9640 1 -- 172.21.15.38:0/2626661059 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f575c011770 con 0x56134b3f5c60 2024-04-22T01:15:19.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.095+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56134b362aa0 con 0x56134b3f5c60 2024-04-22T01:15:19.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.101+0000 7f5754ff9640 1 -- 172.21.15.38:0/2626661059 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f575c011970 con 0x56134b3f5c60 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.102+0000 7f57761ee740 10 client.25796 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:19.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.102+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56134b46c900 con 0x56134b3f5c60 2024-04-22T01:15:19.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.111+0000 7f5754ff9640 1 -- 172.21.15.38:0/2626661059 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56134b362aa0 con 0x56134b3f5c60 2024-04-22T01:15:19.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.111+0000 7f5754ff9640 1 -- 172.21.15.38:0/2626661059 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f575c0253a0 con 0x56134b3f5c60 2024-04-22T01:15:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.113+0000 7f57761ee740 10 client.25796 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.113+0000 7f57761ee740 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.114+0000 7f57761ee740 20 client.25796 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.114+0000 7f57761ee740 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b46d220 0x56134b46f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.114+0000 7f57761ee740 4 client.25796 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:15:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.114+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56134b35cdf0 con 0x56134b46d220 2024-04-22T01:15:19.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.119+0000 7f577139d640 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b46d220 0x56134b46f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.135+0000 7f577139d640 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b46d220 0x56134b46f640 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c4f7604585cc97e server_cookie=55d029705361eaa4 in_seq=0 out_seq=0 2024-04-22T01:15:19.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.137+0000 7f5754ff9640 10 client.25796 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:19.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.140+0000 7f5754ff9640 1 -- 172.21.15.38:0/2626661059 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56134b35cdf0 con 0x56134b46d220 2024-04-22T01:15:19.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.140+0000 7f5754ff9640 10 client.25796 handle_command_reply: tid=0 2024-04-22T01:15:19.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.140+0000 7f57761ee740 1 client.25796 shutdown 2024-04-22T01:15:19.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.146+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 msgr2=0x56134b3ff610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.146+0000 7f57761ee740 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3ff610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f575c010010 tx=0x7f575c0138f0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.149+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 shutdown_connections 2024-04-22T01:15:19.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.149+0000 7f57761ee740 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x56134b46d220 0x56134b46f640 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.149+0000 7f57761ee740 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56134b400620 0x56134b400aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.149+0000 7f57761ee740 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56134b3f5c60 0x56134b3ff610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.149+0000 7f57761ee740 1 --2- 172.21.15.38:0/2626661059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56134b3f0af0 0x56134b3ff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.149+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 >> 172.21.15.38:0/2626661059 conn(0x56134b3eb670 msgr2=0x56134b3f1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.150+0000 7f5754ff9640 0 client.25796 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:19.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.152+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 shutdown_connections 2024-04-22T01:15:19.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.152+0000 7f57761ee740 1 -- 172.21.15.38:0/2626661059 wait complete. 2024-04-22T01:15:19.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.152+0000 7f57761ee740 20 client.25796 trim_cache size 0 max 16384 2024-04-22T01:15:19.155 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:15:19.156 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T01:15:19.156 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T01:15:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f57580a2d10 msgr2=0x7f57580a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f57580a2d10 0x7f57580a51d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f576803b000 tx=0x7f5768048000 comp rx=0 tx=0).stop 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 msgr2=0x7f576413af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 0x7f576413af30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f57640a6030 tx=0x7f576c06d900 comp rx=0 tx=0).stop 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 shutdown_connections 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f57580a2d10 0x7f57580a51d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57640a6b90 0x7f57641424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57640a61d0 0x7f576413b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 --2- 172.21.15.38:0/3164392114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57640a4fe0 0x7f576413af30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.160+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 >> 172.21.15.38:0/3164392114 conn(0x7f57640a0530 msgr2=0x7f57640a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.161+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 shutdown_connections 2024-04-22T01:15:19.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.161+0000 7f577239f640 1 -- 172.21.15.38:0/3164392114 wait complete. 2024-04-22T01:15:19.199 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T01:15:19.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:19.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.446+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3711803924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c0810dd80 msgr2=0x7f8c08110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.446+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3711803924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c0810dd80 0x7f8c08110170 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009930 tx=0x7f8bf802bc60 comp rx=0 tx=0).stop 2024-04-22T01:15:19.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.447+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3711803924 shutdown_connections 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.447+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3711803924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c0810dd80 0x7f8c08110170 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.447+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3711803924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c080741a0 0x7f8c08074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.447+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3711803924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c08075780 0x7f8c08073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.447+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3711803924 >> 172.21.15.38:0/3711803924 conn(0x7f8c080fcd40 msgr2=0x7f8c080ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.448+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3711803924 shutdown_connections 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.448+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3711803924 wait complete. 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.450+0000 7f8c0cd3a640 1 Processor -- start 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.450+0000 7f8c0cd3a640 1 -- start start 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.450+0000 7f8c0cd3a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c080741a0 0x7f8c08072440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.451+0000 7f8c0cd3a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 0x7f8c08072980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.451+0000 7f8c0cd3a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c0810dd80 0x7f8c081adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.451+0000 7f8c0cd3a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0806c9e0 con 0x7f8c08075780 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.451+0000 7f8c0cd3a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0806c860 con 0x7f8c0810dd80 2024-04-22T01:15:19.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.451+0000 7f8c0cd3a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0806cb60 con 0x7f8c080741a0 2024-04-22T01:15:19.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.453+0000 7f8c05d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 0x7f8c08072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.453+0000 7f8c05d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 0x7f8c08072980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32882/0 (socket says 172.21.15.38:32882) 2024-04-22T01:15:19.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.453+0000 7f8c05d9b640 1 -- 172.21.15.38:0/3403179184 learned_addr learned my addr 172.21.15.38:0/3403179184 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:19.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.454+0000 7f8c06d9d640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c0810dd80 0x7f8c081adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.455+0000 7f8c0659c640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c080741a0 0x7f8c08072440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.457+0000 7f8c05d9b640 1 -- 172.21.15.38:0/3403179184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c080741a0 msgr2=0x7f8c08072440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.457+0000 7f8c05d9b640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c080741a0 0x7f8c08072440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.457+0000 7f8c05d9b640 1 -- 172.21.15.38:0/3403179184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c0810dd80 msgr2=0x7f8c081adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:19.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.457+0000 7f8c05d9b640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c0810dd80 0x7f8c081adff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:19.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.458+0000 7f8c05d9b640 1 -- 172.21.15.38:0/3403179184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c081055f0 con 0x7f8c08075780 2024-04-22T01:15:19.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.461+0000 7f8c05d9b640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 0x7f8c08072980 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00b4d0 tx=0x7f8bfc00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:19.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.465+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc014070 con 0x7f8c08075780 2024-04-22T01:15:19.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.465+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bfc010d50 con 0x7f8c08075780 2024-04-22T01:15:19.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.465+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc017360 con 0x7f8c08075780 2024-04-22T01:15:19.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.466+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c081058e0 con 0x7f8c08075780 2024-04-22T01:15:19.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.466+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c08105db0 con 0x7f8c08075780 2024-04-22T01:15:19.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.472+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be80052c0 con 0x7f8c08075780 2024-04-22T01:15:19.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.476+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8bfc017070 con 0x7f8c08075780 2024-04-22T01:15:19.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.477+0000 7f8bf77fe640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8bf00a2d10 0x7f8bf00a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:19.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.479+0000 7f8c0659c640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8bf00a2d10 0x7f8bf00a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:19.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.483+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f8bfc098300 con 0x7f8c08075780 2024-04-22T01:15:19.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.489+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8bfc09a960 con 0x7f8c08075780 2024-04-22T01:15:19.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:19.493+0000 7f8c0659c640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8bf00a2d10 0x7f8bf00a51d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009fd0 tx=0x7f8c0000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:19 smithi203 ceph-mon[31825]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 710 MiB/s wr, 289 op/s 2024-04-22T01:15:20.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.066+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8be80055b0 con 0x7f8c08075780 2024-04-22T01:15:20.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.068+0000 7f8bf77fe640 1 -- 172.21.15.38:0/3403179184 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f8bfc09a260 con 0x7f8c08075780 2024-04-22T01:15:20.068 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:15:20.069 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:15:20.069 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:15:20.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.073+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8bf00a2d10 msgr2=0x7f8bf00a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.073+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8bf00a2d10 0x7f8bf00a51d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009fd0 tx=0x7f8c0000d040 comp rx=0 tx=0).stop 2024-04-22T01:15:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.073+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 msgr2=0x7f8c08072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.073+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 0x7f8c08072980 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00b4d0 tx=0x7f8bfc00b990 comp rx=0 tx=0).stop 2024-04-22T01:15:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.074+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 shutdown_connections 2024-04-22T01:15:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.074+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f8bf00a2d10 0x7f8bf00a51d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.074+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c0810dd80 0x7f8c081adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.074+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c08075780 0x7f8c08072980 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.074+0000 7f8c0cd3a640 1 --2- 172.21.15.38:0/3403179184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c080741a0 0x7f8c08072440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.074+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 >> 172.21.15.38:0/3403179184 conn(0x7f8c080fcd40 msgr2=0x7f8c08101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.075+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 shutdown_connections 2024-04-22T01:15:20.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.075+0000 7f8c0cd3a640 1 -- 172.21.15.38:0/3403179184 wait complete. 2024-04-22T01:15:20.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T01:15:20.106 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:19 smithi038 ceph-mon[28043]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 710 MiB/s wr, 289 op/s 2024-04-22T01:15:20.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:19 smithi111 ceph-mon[31057]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 710 MiB/s wr, 289 op/s 2024-04-22T01:15:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.329+0000 7f6c96cdf640 1 -- 172.21.15.38:0/329239953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90100000 msgr2=0x7f6c90102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.329+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/329239953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90100000 0x7f6c90102430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009900 tx=0x7f6c7802bc40 comp rx=0 tx=0).stop 2024-04-22T01:15:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.330+0000 7f6c96cdf640 1 -- 172.21.15.38:0/329239953 shutdown_connections 2024-04-22T01:15:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.330+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/329239953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 0x7f6c901057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.330+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/329239953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90102970 0x7f6c90104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.330+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/329239953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90100000 0x7f6c90102430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.330+0000 7f6c96cdf640 1 -- 172.21.15.38:0/329239953 >> 172.21.15.38:0/329239953 conn(0x7f6c900fbc00 msgr2=0x7f6c900fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.332+0000 7f6c96cdf640 1 -- 172.21.15.38:0/329239953 shutdown_connections 2024-04-22T01:15:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.332+0000 7f6c96cdf640 1 -- 172.21.15.38:0/329239953 wait complete. 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.333+0000 7f6c96cdf640 1 Processor -- start 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.333+0000 7f6c96cdf640 1 -- start start 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.333+0000 7f6c96cdf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90100000 0x7f6c9019e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.333+0000 7f6c96cdf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90102970 0x7f6c9019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c96cdf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 0x7f6c901a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c96cdf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6c90113a30 con 0x7f6c90105320 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c96cdf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901138b0 con 0x7f6c90100000 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c96cdf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c90113bb0 con 0x7f6c90102970 2024-04-22T01:15:20.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c94a7b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90100000 0x7f6c9019e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c9527c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 0x7f6c901a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c94a7b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90100000 0x7f6c9019e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55494/0 (socket says 172.21.15.38:55494) 2024-04-22T01:15:20.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c94a7b640 1 -- 172.21.15.38:0/2025449476 learned_addr learned my addr 172.21.15.38:0/2025449476 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.334+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90102970 0x7f6c9019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.335+0000 7f6c9527c640 1 -- 172.21.15.38:0/2025449476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90102970 msgr2=0x7f6c9019e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.335+0000 7f6c9527c640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90102970 0x7f6c9019e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.335+0000 7f6c9527c640 1 -- 172.21.15.38:0/2025449476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90100000 msgr2=0x7f6c9019e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.335+0000 7f6c9527c640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90100000 0x7f6c9019e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.335+0000 7f6c9527c640 1 -- 172.21.15.38:0/2025449476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c901a5ca0 con 0x7f6c90105320 2024-04-22T01:15:20.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.336+0000 7f6c9527c640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 0x7f6c901a5630 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800e940 tx=0x7f6c8800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.337+0000 7f6c8dffb640 1 -- 172.21.15.38:0/2025449476 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c88004d20 con 0x7f6c90105320 2024-04-22T01:15:20.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.337+0000 7f6c8dffb640 1 -- 172.21.15.38:0/2025449476 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c88005020 con 0x7f6c90105320 2024-04-22T01:15:20.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.337+0000 7f6c8dffb640 1 -- 172.21.15.38:0/2025449476 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880116b0 con 0x7f6c90105320 2024-04-22T01:15:20.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.337+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c901a5f90 con 0x7f6c90105320 2024-04-22T01:15:20.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.337+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c901a6400 con 0x7f6c90105320 2024-04-22T01:15:20.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.342+0000 7f6c999a4740 1 Processor -- start 2024-04-22T01:15:20.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.343+0000 7f6c8dffb640 1 -- 172.21.15.38:0/2025449476 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f6c8800cb20 con 0x7f6c90105320 2024-04-22T01:15:20.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 -- start start 2024-04-22T01:15:20.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539b9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539be530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 0x555a539c1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555a5391e530 con 0x555a539beba0 2024-04-22T01:15:20.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555a538c6830 con 0x555a539b9040 2024-04-22T01:15:20.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.344+0000 7f6c999a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555a5391a4f0 con 0x555a539b9a30 2024-04-22T01:15:20.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.345+0000 7f6c8dffb640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c7c0a2ce0 0x7f6c7c0a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.345+0000 7f6c8dffb640 1 -- 172.21.15.38:0/2025449476 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6c880cf3b0 con 0x7f6c90105320 2024-04-22T01:15:20.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.346+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 0x555a539c1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.346+0000 7f6c9527c640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c7c0a2ce0 0x7f6c7c0a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.346+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 0x555a539c1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32918/0 (socket says 172.21.15.38:32918) 2024-04-22T01:15:20.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.346+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1814899933 learned_addr learned my addr 172.21.15.38:0/1814899933 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.346+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.347+0000 7f6c9527c640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c7c0a2ce0 0x7f6c7c0a51a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800efd0 tx=0x7f6c88004f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.347+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1814899933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 msgr2=0x555a539be530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.347+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539be530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.347+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1814899933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 msgr2=0x555a539b9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.347+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539b9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.347+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1814899933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a538ea2c0 con 0x555a539beba0 2024-04-22T01:15:20.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.348+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 0x555a539c1030 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ce30 tx=0x7f6c8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3347029d28ddd047 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.348+0000 7f6c86ffd640 1 -- 172.21.15.38:0/1814899933 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c80005020 con 0x555a539beba0 2024-04-22T01:15:20.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.348+0000 7f6c86ffd640 1 -- 172.21.15.38:0/1814899933 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c80016030 con 0x555a539beba0 2024-04-22T01:15:20.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.348+0000 7f6c86ffd640 1 -- 172.21.15.38:0/1814899933 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8000f950 con 0x555a539beba0 2024-04-22T01:15:20.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 -- 172.21.15.38:0/1814899933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 msgr2=0x555a539c1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 0x555a539c1030 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ce30 tx=0x7f6c8000c3a0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 -- 172.21.15.38:0/1814899933 shutdown_connections 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539beba0 0x555a539c1030 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539be530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1814899933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539b9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 -- 172.21.15.38:0/1814899933 >> 172.21.15.38:0/1814899933 conn(0x555a539b45b0 msgr2=0x555a539b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.349+0000 7f6c999a4740 1 -- 172.21.15.38:0/1814899933 shutdown_connections 2024-04-22T01:15:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.353+0000 7f6c999a4740 1 -- 172.21.15.38:0/1814899933 wait complete. 2024-04-22T01:15:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.353+0000 7f6c999a4740 1 -- start start 2024-04-22T01:15:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 Processor -- start 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 0x555a539c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9a30 0x555a539c3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555a539c1b50 con 0x555a539b9040 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555a538d9f10 con 0x555a539b9a30 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555a539c1cd0 con 0x555a539c88c0 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.354+0000 7f6c999a4740 10 client.0 _ll_register_callbacks cb 0x555a53631be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.355+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 0x555a539c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 0x555a539c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32924/0 (socket says 172.21.15.38:32924) 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 -- 172.21.15.38:0/675492436 learned_addr learned my addr 172.21.15.38:0/675492436 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9a30 0x555a539c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 -- 172.21.15.38:0/675492436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 msgr2=0x555a539cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 -- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9a30 msgr2=0x555a539c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9a30 0x555a539c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c8ffff640 1 -- 172.21.15.38:0/675492436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a538abc80 con 0x555a539b9040 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.356+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9a30 0x555a539c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.357+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 0x555a539c3350 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6c80005d50 tx=0x7f6c80005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78bfaa2efbb49b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.357+0000 7f6c6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:20.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.359+0000 7f6c6ffff640 1 -- 172.21.15.38:0/675492436 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c80005210 con 0x555a539b9040 2024-04-22T01:15:20.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.359+0000 7f6c6ffff640 1 -- 172.21.15.38:0/675492436 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6c800053b0 con 0x555a539b9040 2024-04-22T01:15:20.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.360+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c80016030 con 0x555a539b9040 2024-04-22T01:15:20.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.360+0000 7f6c6ffff640 1 -- 172.21.15.38:0/675492436 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6c8002f050 con 0x555a539b9040 2024-04-22T01:15:20.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.361+0000 7f6c999a4740 10 client.16314 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.361+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555a536323f0 con 0x555a539b9040 2024-04-22T01:15:20.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c6ffff640 1 -- 172.21.15.38:0/675492436 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c80016030 con 0x555a539b9040 2024-04-22T01:15:20.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c6ffff640 1 -- 172.21.15.38:0/675492436 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6c8000fb80 con 0x555a539b9040 2024-04-22T01:15:20.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c999a4740 10 client.16314 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c999a4740 10 client.16314 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:15:20.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c999a4740 20 client.16314 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c999a4740 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a539e0210 0x555a539e2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c999a4740 4 client.16314 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:20.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.362+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a5392ba70 con 0x555a539e0210 2024-04-22T01:15:20.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.363+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a539e0210 0x555a539e2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.366+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a539e0210 0x555a539e2600 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a61e5d970eae0a3c server_cookie=482062dd6bf53d1c in_seq=0 out_seq=0 2024-04-22T01:15:20.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.366+0000 7f6c6ffff640 10 client.16314 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:20.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.367+0000 7f6c6ffff640 1 -- 172.21.15.38:0/675492436 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a5392ba70 con 0x555a539e0210 2024-04-22T01:15:20.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.367+0000 7f6c6ffff640 10 client.16314 handle_command_reply: tid=0 2024-04-22T01:15:20.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.367+0000 7f6c999a4740 1 client.16314 shutdown 2024-04-22T01:15:20.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.373+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 msgr2=0x555a539c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.373+0000 7f6c999a4740 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 0x555a539c3350 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6c80005d50 tx=0x7f6c80005ce0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.375+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 shutdown_connections 2024-04-22T01:15:20.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.375+0000 7f6c999a4740 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a539e0210 0x555a539e2600 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.375+0000 7f6c999a4740 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.375+0000 7f6c999a4740 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9a30 0x555a539c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.375+0000 7f6c999a4740 1 --2- 172.21.15.38:0/675492436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9040 0x555a539c3350 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.375+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 >> 172.21.15.38:0/675492436 conn(0x555a539b45b0 msgr2=0x555a539b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.376+0000 7f6c6ffff640 0 client.16314 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:20.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.378+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 shutdown_connections 2024-04-22T01:15:20.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.378+0000 7f6c999a4740 1 -- 172.21.15.38:0/675492436 wait complete. 2024-04-22T01:15:20.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.378+0000 7f6c999a4740 20 client.16314 trim_cache size 0 max 16384 2024-04-22T01:15:20.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.411+0000 7f6c999a4740 1 Processor -- start 2024-04-22T01:15:20.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.413+0000 7f6c999a4740 1 -- start start 2024-04-22T01:15:20.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.413+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.413+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.413+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55522/0 (socket says 172.21.15.38:55522) 2024-04-22T01:15:20.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c8ffff640 1 -- 172.21.15.38:0/2551284964 learned_addr learned my addr 172.21.15.38:0/2551284964 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c999a4740 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539c88c0 0x555a539ba120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7000fdb0 con 0x555a539c88c0 2024-04-22T01:15:20.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6c700118f0 con 0x555a539b9040 2024-04-22T01:15:20.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c70010480 con 0x555a539b9a30 2024-04-22T01:15:20.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.414+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539c88c0 0x555a539ba120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 -- 172.21.15.38:0/2551284964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 msgr2=0x555a539b7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539b7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 -- 172.21.15.38:0/2551284964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539c88c0 msgr2=0x555a539ba120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539c88c0 0x555a539ba120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 -- 172.21.15.38:0/2551284964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a539cb020 con 0x555a539b9040 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.415+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539c37f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ff50 tx=0x7f6c800035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b01b5c20ead7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.416+0000 7f6c867fc640 1 -- 172.21.15.38:0/2551284964 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c80004660 con 0x555a539b9040 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.416+0000 7f6c867fc640 1 -- 172.21.15.38:0/2551284964 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c80016070 con 0x555a539b9040 2024-04-22T01:15:20.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.416+0000 7f6c867fc640 1 -- 172.21.15.38:0/2551284964 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8000f660 con 0x555a539b9040 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 msgr2=0x555a539c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539c37f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ff50 tx=0x7f6c800035f0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 shutdown_connections 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539c88c0 0x555a539ba120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539b9a30 0x555a539b7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 --2- 172.21.15.38:0/2551284964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539b9040 0x555a539c37f0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ff50 tx=0x7f6c800035f0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.417+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 >> 172.21.15.38:0/2551284964 conn(0x555a539b45b0 msgr2=0x555a539b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.419+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 shutdown_connections 2024-04-22T01:15:20.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.419+0000 7f6c999a4740 1 -- 172.21.15.38:0/2551284964 wait complete. 2024-04-22T01:15:20.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.420+0000 7f6c999a4740 1 -- start start 2024-04-22T01:15:20.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.420+0000 7f6c999a4740 1 Processor -- start 2024-04-22T01:15:20.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.421+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9a30 0x555a539f1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.421+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c999a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 0x555a539f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c999a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6c70011220 con 0x555a539b9a30 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 0x555a539f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 0x555a539f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55532/0 (socket says 172.21.15.38:55532) 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1123528856 learned_addr learned my addr 172.21.15.38:0/1123528856 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6c70010b50 con 0x555a539f2d50 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7000dd70 con 0x555a539c88c0 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c999a4740 10 client.0 _ll_register_callbacks cb 0x7f6c8000cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.422+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9a30 0x555a539f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.423+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1123528856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 msgr2=0x555a539f1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.423+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539f1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.423+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1123528856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9a30 msgr2=0x555a539f1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.423+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9a30 0x555a539f1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.423+0000 7f6c8ffff640 1 -- 172.21.15.38:0/1123528856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a536fc7d0 con 0x555a539f2d50 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c8ffff640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 0x555a539f31d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x555a539c4680 tx=0x7f6c80001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19f9164e786afa7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c6ffff640 1 -- 172.21.15.38:0/1123528856 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c80004ac0 con 0x555a539f2d50 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c80016070 con 0x555a539f2d50 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c6ffff640 1 -- 172.21.15.38:0/1123528856 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f6c80004c60 con 0x555a539f2d50 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c6ffff640 1 -- 172.21.15.38:0/1123528856 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6c80004e60 con 0x555a539f2d50 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c999a4740 10 client.25808 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.424+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555a53a46ad0 con 0x555a539f2d50 2024-04-22T01:15:20.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c6ffff640 1 -- 172.21.15.38:0/1123528856 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c80016070 con 0x555a539f2d50 2024-04-22T01:15:20.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c6ffff640 1 -- 172.21.15.38:0/1123528856 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6c8001b750 con 0x555a539f2d50 2024-04-22T01:15:20.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c999a4740 10 client.25808 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c999a4740 10 client.25808 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:15:20.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c999a4740 20 client.25808 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a53a47380 0x555a53a497a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c999a4740 4 client.25808 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:20.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.425+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a53925dc0 con 0x555a53a47380 2024-04-22T01:15:20.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.426+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a53a47380 0x555a53a497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.427+0000 7f6c94a7b640 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a53a47380 0x555a53a497a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d029ab1c7e9e71c0 server_cookie=9a264349a380bb0e in_seq=0 out_seq=0 2024-04-22T01:15:20.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.427+0000 7f6c6ffff640 10 client.25808 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.427+0000 7f6c6ffff640 1 -- 172.21.15.38:0/1123528856 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a53925dc0 con 0x555a53a47380 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.427+0000 7f6c6ffff640 10 client.25808 handle_command_reply: tid=0 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.427+0000 7f6c999a4740 1 client.25808 shutdown 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.427+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 msgr2=0x555a539f31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 0x555a539f31d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x555a539c4680 tx=0x7f6c80001e50 comp rx=0 tx=0).stop 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 shutdown_connections 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x555a53a47380 0x555a53a497a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a539f2d50 0x555a539f31d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a539c88c0 0x555a539f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 --2- 172.21.15.38:0/1123528856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a539b9a30 0x555a539f1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c6ffff640 0 client.25808 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 >> 172.21.15.38:0/1123528856 conn(0x555a539b45b0 msgr2=0x555a539b80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 shutdown_connections 2024-04-22T01:15:20.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 1 -- 172.21.15.38:0/1123528856 wait complete. 2024-04-22T01:15:20.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.428+0000 7f6c999a4740 20 client.25808 trim_cache size 0 max 16384 2024-04-22T01:15:20.430 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:20.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.430+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c7c0a2ce0 msgr2=0x7f6c7c0a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.430+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c7c0a2ce0 0x7f6c7c0a51a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800efd0 tx=0x7f6c88004f60 comp rx=0 tx=0).stop 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.430+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 msgr2=0x7f6c901a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.430+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 0x7f6c901a5630 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800e940 tx=0x7f6c8800ee00 comp rx=0 tx=0).stop 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.431+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 shutdown_connections 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.431+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f6c7c0a2ce0 0x7f6c7c0a51a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.431+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105320 0x7f6c901a5630 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.431+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c90102970 0x7f6c9019e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.431+0000 7f6c96cdf640 1 --2- 172.21.15.38:0/2025449476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c90100000 0x7f6c9019e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.431+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 >> 172.21.15.38:0/2025449476 conn(0x7f6c900fbc00 msgr2=0x7f6c901033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.432+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 shutdown_connections 2024-04-22T01:15:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.432+0000 7f6c96cdf640 1 -- 172.21.15.38:0/2025449476 wait complete. 2024-04-22T01:15:20.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 rewriters = 957814.81 kB/sec 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 rewriters = 957797.93 kB/sec 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 957814.81 kB/sec 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 957814.81 kB/sec 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 957814.81 kB/sec 2024-04-22T01:15:20.568 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 10485760.00 kB 2024-04-22T01:15:20.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 -- 172.21.15.38:0/1313413765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 msgr2=0x7f48c0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 --2- 172.21.15.38:0/1313413765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c0108ae0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009ca0 tx=0x7f48b802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 -- 172.21.15.38:0/1313413765 shutdown_connections 2024-04-22T01:15:20.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 --2- 172.21.15.38:0/1313413765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c0108ae0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 --2- 172.21.15.38:0/1313413765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c0101890 0x7f48c0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 --2- 172.21.15.38:0/1313413765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c01006a0 0x7f48c0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 -- 172.21.15.38:0/1313413765 >> 172.21.15.38:0/1313413765 conn(0x7f48c00fbc30 msgr2=0x7f48c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 -- 172.21.15.38:0/1313413765 shutdown_connections 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 -- 172.21.15.38:0/1313413765 wait complete. 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 Processor -- start 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.599+0000 7f48c6078640 1 -- start start 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48c6078640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c01006a0 0x7f48c006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48c6078640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c0101890 0x7f48c019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48c6078640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48c6078640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48c010c300 con 0x7f48c0102250 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48c6078640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48c010c180 con 0x7f48c01006a0 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48c6078640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48c010c480 con 0x7f48c0101890 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c019ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32954/0 (socket says 172.21.15.38:32954) 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 -- 172.21.15.38:0/4212423636 learned_addr learned my addr 172.21.15.38:0/4212423636 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c01006a0 0x7f48c006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48beffd640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c0101890 0x7f48c019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 -- 172.21.15.38:0/4212423636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c0101890 msgr2=0x7f48c019c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c0101890 0x7f48c019c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 -- 172.21.15.38:0/4212423636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c01006a0 msgr2=0x7f48c006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c01006a0 0x7f48c006a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 -- 172.21.15.38:0/4212423636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c019f6a0 con 0x7f48c0102250 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c01006a0 0x7f48c006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48beffd640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c0101890 0x7f48c019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bffff640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c019ede0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f48b8037040 tx=0x7f48b8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.600+0000 7f48bcff9640 1 -- 172.21.15.38:0/4212423636 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8041020 con 0x7f48c0102250 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.601+0000 7f48bcff9640 1 -- 172.21.15.38:0/4212423636 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b8034d90 con 0x7f48c0102250 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.601+0000 7f48bcff9640 1 -- 172.21.15.38:0/4212423636 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8039040 con 0x7f48c0102250 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.601+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c019f930 con 0x7f48c0102250 2024-04-22T01:15:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.601+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c019fe20 con 0x7f48c0102250 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48bcff9640 1 -- 172.21.15.38:0/4212423636 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f48b8044020 con 0x7f48c0102250 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 Processor -- start 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 -- start start 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48bcff9640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48a80a2b90 0x7f48a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da03870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da08940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da0b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564c5d964530 con 0x564c5da08fb0 2024-04-22T01:15:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564c5d90c830 con 0x564c5da03450 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.603+0000 7f48c8c59740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564c5d9604f0 con 0x564c5da03e40 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bcff9640 1 -- 172.21.15.38:0/4212423636 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f48b80f13a0 con 0x7f48c0102250 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48beffd640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48a80a2b90 0x7f48a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da0b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da0b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32958/0 (socket says 172.21.15.38:32958) 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da03870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bffff640 1 -- 172.21.15.38:0/789486233 learned_addr learned my addr 172.21.15.38:0/789486233 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48beffd640 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da08940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 -- 172.21.15.38:0/789486233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 msgr2=0x564c5da08940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da08940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 -- 172.21.15.38:0/789486233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 msgr2=0x564c5da0b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da0b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 -- 172.21.15.38:0/789486233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c5d9302c0 con 0x564c5da03450 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bffff640 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da0b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48beffd640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48a80a2b90 0x7f48a80a5050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f48c006a640 tx=0x7f48b40147e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da03870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48c01016f0 tx=0x7f48b00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8857284082ccfb8a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48a5ffb640 1 -- 172.21.15.38:0/789486233 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0005220 con 0x564c5da03450 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48a5ffb640 1 -- 172.21.15.38:0/789486233 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b0004c80 con 0x564c5da03450 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.604+0000 7f48a5ffb640 1 -- 172.21.15.38:0/789486233 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0011960 con 0x564c5da03450 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 -- 172.21.15.38:0/789486233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 msgr2=0x564c5da03870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da03870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48c01016f0 tx=0x7f48b00107a0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 -- 172.21.15.38:0/789486233 shutdown_connections 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da0b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da08940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 --2- 172.21.15.38:0/789486233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da03870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 -- 172.21.15.38:0/789486233 >> 172.21.15.38:0/789486233 conn(0x564c5d9fa5b0 msgr2=0x564c5d9fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 -- 172.21.15.38:0/789486233 shutdown_connections 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 -- 172.21.15.38:0/789486233 wait complete. 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.605+0000 7f48c8c59740 1 -- start start 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 Processor -- start 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da0d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da0dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da14c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564c5da0bf60 con 0x564c5da08fb0 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564c5d91ff10 con 0x564c5da03450 2024-04-22T01:15:20.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564c5da0c0e0 con 0x564c5da03e40 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48c8c59740 10 client.0 _ll_register_callbacks cb 0x564c5d677be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da14c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da0d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da14c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32968/0 (socket says 172.21.15.38:32968) 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da0d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55574/0 (socket says 172.21.15.38:55574) 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 -- 172.21.15.38:0/1404583538 learned_addr learned my addr 172.21.15.38:0/1404583538 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48beffd640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da0dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 -- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 msgr2=0x564c5da0dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da0dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 -- 172.21.15.38:0/1404583538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 msgr2=0x564c5da0d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da0d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 -- 172.21.15.38:0/1404583538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c5d8f1c80 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da0d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.606+0000 7f48bffff640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da14c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f48b8003d50 tx=0x7f48b80bd370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5638655e482e07db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f489effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f489effd640 1 -- 172.21.15.38:0/1404583538 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8005610 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f489effd640 1 -- 172.21.15.38:0/1404583538 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f48b80a80c0 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b0004c80 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f489effd640 1 -- 172.21.15.38:0/1404583538 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48b8005430 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 10 client.16332 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c5d6783f0 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f489effd640 1 -- 172.21.15.38:0/1404583538 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b0004c80 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f489effd640 1 -- 172.21.15.38:0/1404583538 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f48b803c030 con 0x564c5da08fb0 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 10 client.16332 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 10 client.16332 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:15:20.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 20 client.16332 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da19480 0x564c5da1b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 4 client.16332 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.607+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c5d971a70 con 0x564c5da19480 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.608+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da19480 0x564c5da1b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.608+0000 7f489effd640 10 client.16332 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.608+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da19480 0x564c5da1b900 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7bd3bb0aa3ec7971 server_cookie=fc646693d58737cf in_seq=0 out_seq=0 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f489effd640 1 -- 172.21.15.38:0/1404583538 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c5d971a70 con 0x564c5da19480 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f489effd640 10 client.16332 handle_command_reply: tid=0 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 client.16332 shutdown 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 msgr2=0x564c5da14c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da14c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f48b8003d50 tx=0x7f48b80bd370 comp rx=0 tx=0).stop 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 shutdown_connections 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da19480 0x564c5da1b900 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da08fb0 0x564c5da14c10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da0dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 --2- 172.21.15.38:0/1404583538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da03450 0x564c5da0d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 >> 172.21.15.38:0/1404583538 conn(0x564c5d9fa5b0 msgr2=0x564c5d9f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f489effd640 0 client.16332 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 shutdown_connections 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 1 -- 172.21.15.38:0/1404583538 wait complete. 2024-04-22T01:15:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.609+0000 7f48c8c59740 20 client.16332 trim_cache size 0 max 16384 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 Processor -- start 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 -- start start 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 0x564c5d9f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da14bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5d9f9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01a8360 con 0x564c5da03450 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01a9ea0 con 0x564c5da08fb0 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.622+0000 7f48c8c59740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01a8a30 con 0x564c5da03e40 2024-04-22T01:15:20.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 0x564c5d9f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 0x564c5d9f8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32982/0 (socket says 172.21.15.38:32982) 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 -- 172.21.15.38:0/2629113716 learned_addr learned my addr 172.21.15.38:0/2629113716 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48beffd640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da14bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bffff640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5d9f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 -- 172.21.15.38:0/2629113716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 msgr2=0x564c5da14bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da14bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 -- 172.21.15.38:0/2629113716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 msgr2=0x564c5d9f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5d9f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 -- 172.21.15.38:0/2629113716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c5d7427d0 con 0x564c5da03450 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48beffd640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da14bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bffff640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5d9f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 0x564c5d9f8fc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f48b0000c00 tx=0x7f48b0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82219405cc75eb1d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48a57fa640 1 -- 172.21.15.38:0/2629113716 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0004220 con 0x564c5da03450 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48a57fa640 1 -- 172.21.15.38:0/2629113716 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b0004dc0 con 0x564c5da03450 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.623+0000 7f48a57fa640 1 -- 172.21.15.38:0/2629113716 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0005260 con 0x564c5da03450 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 -- 172.21.15.38:0/2629113716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 msgr2=0x564c5d9f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 0x564c5d9f8fc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f48b0000c00 tx=0x7f48b0005850 comp rx=0 tx=0).stop 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 -- 172.21.15.38:0/2629113716 shutdown_connections 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5d9f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03e40 0x564c5da14bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 --2- 172.21.15.38:0/2629113716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03450 0x564c5d9f8fc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 -- 172.21.15.38:0/2629113716 >> 172.21.15.38:0/2629113716 conn(0x564c5d9fa5b0 msgr2=0x564c5da0d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 -- 172.21.15.38:0/2629113716 shutdown_connections 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 -- 172.21.15.38:0/2629113716 wait complete. 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 -- start start 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.624+0000 7f48c8c59740 1 Processor -- start 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03450 0x564c5da14a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 0x564c5da11510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5da11a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01a97d0 con 0x564c5da03e40 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01a9100 con 0x564c5da08fb0 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48c01a6320 con 0x564c5da03450 2024-04-22T01:15:20.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 10 client.0 _ll_register_callbacks cb 0x7f48b80eea10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 0x564c5da11510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 0x564c5da11510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32990/0 (socket says 172.21.15.38:32990) 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 -- 172.21.15.38:0/3050969776 learned_addr learned my addr 172.21.15.38:0/3050969776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48bffff640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5da11a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03450 0x564c5da14a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 -- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03450 msgr2=0x564c5da14a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03450 0x564c5da14a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 -- 172.21.15.38:0/3050969776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 msgr2=0x564c5da11a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5da11a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 -- 172.21.15.38:0/3050969776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c5da7da70 con 0x564c5da03e40 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48bffff640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5da11a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48beffd640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 0x564c5da11510 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f48b4016010 tx=0x7f48b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bf6c963fe87704 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f489effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:20.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f489effd640 1 -- 172.21.15.38:0/3050969776 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b402a640 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f489effd640 1 -- 172.21.15.38:0/3050969776 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f48b4020440 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.625+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564c5d96bdc0 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f489effd640 1 -- 172.21.15.38:0/3050969776 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48b402a7e0 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 10 client.16338 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c5da7ecc0 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f489effd640 1 -- 172.21.15.38:0/3050969776 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564c5d96bdc0 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f489effd640 1 -- 172.21.15.38:0/3050969776 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f48b4028070 con 0x564c5da03e40 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 10 client.16338 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 10 client.16338 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 20 client.16338 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da7f720 0x564c5da81b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 4 client.16338 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c5d8e96e0 con 0x564c5da7f720 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.626+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da7f720 0x564c5da81b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48bf7fe640 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da7f720 0x564c5da81b40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15e99b9f6029f2bc server_cookie=4c7a136d133bb98d in_seq=0 out_seq=0 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f489effd640 10 client.16338 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f489effd640 1 -- 172.21.15.38:0/3050969776 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c5d8e96e0 con 0x564c5da7f720 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f489effd640 10 client.16338 handle_command_reply: tid=0 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 client.16338 shutdown 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 msgr2=0x564c5da11510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 0x564c5da11510 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f48b4016010 tx=0x7f48b4004730 comp rx=0 tx=0).stop 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 shutdown_connections 2024-04-22T01:15:20.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x564c5da7f720 0x564c5da81b40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c5da08fb0 0x564c5da11a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c5da03e40 0x564c5da11510 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f489effd640 0 client.16338 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 --2- 172.21.15.38:0/3050969776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c5da03450 0x564c5da14a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 >> 172.21.15.38:0/3050969776 conn(0x564c5d9fa5b0 msgr2=0x564c5d9fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 shutdown_connections 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 1 -- 172.21.15.38:0/3050969776 wait complete. 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.627+0000 7f48c8c59740 20 client.16338 trim_cache size 0 max 16384 2024-04-22T01:15:20.636 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:20.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48a80a2b90 msgr2=0x7f48a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48a80a2b90 0x7f48a80a5050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f48c006a640 tx=0x7f48b40147e0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 msgr2=0x7f48c019ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c019ede0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f48b8037040 tx=0x7f48b8032f20 comp rx=0 tx=0).stop 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 shutdown_connections 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48a80a2b90 0x7f48a80a5050 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48c0102250 0x7f48c019ede0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48c0101890 0x7f48c019c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 --2- 172.21.15.38:0/4212423636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48c01006a0 0x7f48c006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 >> 172.21.15.38:0/4212423636 conn(0x7f48c00fbc30 msgr2=0x7f48c0104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 shutdown_connections 2024-04-22T01:15:20.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.628+0000 7f48c6078640 1 -- 172.21.15.38:0/4212423636 wait complete. 2024-04-22T01:15:20.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T01:15:20.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 -- 172.21.15.38:0/497170209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b98103350 msgr2=0x7f4b98109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/497170209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b98103350 0x7f4b98109be0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009ca0 tx=0x7f4b8802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 -- 172.21.15.38:0/497170209 shutdown_connections 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/497170209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b98103350 0x7f4b98109be0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/497170209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 0x7f4b98102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/497170209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b981017a0 0x7f4b98101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 -- 172.21.15.38:0/497170209 >> 172.21.15.38:0/497170209 conn(0x7f4b980fcd10 msgr2=0x7f4b980ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 -- 172.21.15.38:0/497170209 shutdown_connections 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 -- 172.21.15.38:0/497170209 wait complete. 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 Processor -- start 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.745+0000 7f4b9fa84640 1 -- start start 2024-04-22T01:15:20.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9fa84640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 0x7f4b980754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9fa84640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 0x7f4b98073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9fa84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98103350 0x7f4b98074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9fa84640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810d400 con 0x7f4b981017a0 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9fa84640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810d280 con 0x7f4b98102990 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9fa84640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810d580 con 0x7f4b98103350 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 0x7f4b980754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 0x7f4b980754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33002/0 (socket says 172.21.15.38:33002) 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 -- 172.21.15.38:0/288509756 learned_addr learned my addr 172.21.15.38:0/288509756 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 0x7f4b98073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9e021640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98103350 0x7f4b98074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 -- 172.21.15.38:0/288509756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98103350 msgr2=0x7f4b98074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98103350 0x7f4b98074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 -- 172.21.15.38:0/288509756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 msgr2=0x7f4b98073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 0x7f4b98073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 -- 172.21.15.38:0/288509756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b98074900 con 0x7f4b981017a0 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 0x7f4b98073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b9d820640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 0x7f4b980754f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000d770 tx=0x7f4b9000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.746+0000 7f4b96ffd640 1 -- 172.21.15.38:0/288509756 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90017070 con 0x7f4b981017a0 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.747+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b981a1b60 con 0x7f4b981017a0 2024-04-22T01:15:20.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.747+0000 7f4b96ffd640 1 -- 172.21.15.38:0/288509756 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b90009c30 con 0x7f4b981017a0 2024-04-22T01:15:20.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.747+0000 7f4b96ffd640 1 -- 172.21.15.38:0/288509756 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90003dd0 con 0x7f4b981017a0 2024-04-22T01:15:20.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.747+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b981a20b0 con 0x7f4b981017a0 2024-04-22T01:15:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.749+0000 7f4b96ffd640 1 -- 172.21.15.38:0/288509756 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4b90003fb0 con 0x7f4b981017a0 2024-04-22T01:15:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.749+0000 7f4ba266e740 1 Processor -- start 2024-04-22T01:15:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.749+0000 7f4ba266e740 1 -- start start 2024-04-22T01:15:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 0x556f581e2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2a30 0x556f581e7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e7ba0 0x556f581ea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4ba266e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556f58147530 con 0x556f581e7ba0 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4ba266e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556f580ef830 con 0x556f581e2040 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4ba266e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556f581434f0 con 0x556f581e2a30 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 0x556f581e2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9d01f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e7ba0 0x556f581ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 0x556f581e2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55612/0 (socket says 172.21.15.38:55612) 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 -- 172.21.15.38:0/1726534237 learned_addr learned my addr 172.21.15.38:0/1726534237 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9d820640 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2a30 0x556f581e7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b96ffd640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4b800a2b90 0x7f4b800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 -- 172.21.15.38:0/1726534237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2a30 msgr2=0x556f581e7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2a30 0x556f581e7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 -- 172.21.15.38:0/1726534237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e7ba0 msgr2=0x556f581ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e7ba0 0x556f581ea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 -- 172.21.15.38:0/1726534237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f581132c0 con 0x556f581e2040 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e7ba0 0x556f581ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b96ffd640 1 -- 172.21.15.38:0/288509756 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4b900cee50 con 0x7f4b981017a0 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4b800a2b90 0x7f4b800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.750+0000 7f4b9e021640 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 0x556f581e2460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b880324c0 tx=0x7f4b88009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57a94ebdfa3d5d70 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4b7ffff640 1 -- 172.21.15.38:0/1726534237 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88039040 con 0x556f581e2040 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4b7ffff640 1 -- 172.21.15.38:0/1726534237 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b88003e50 con 0x556f581e2040 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4b7ffff640 1 -- 172.21.15.38:0/1726534237 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88034440 con 0x556f581e2040 2024-04-22T01:15:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4b9e021640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4b800a2b90 0x7f4b800a5050 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4b8804b000 tx=0x7f4b88044000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 -- 172.21.15.38:0/1726534237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 msgr2=0x556f581e2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 0x556f581e2460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b880324c0 tx=0x7f4b88009800 comp rx=0 tx=0).stop 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 -- 172.21.15.38:0/1726534237 shutdown_connections 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e7ba0 0x556f581ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2a30 0x556f581e7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 --2- 172.21.15.38:0/1726534237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2040 0x556f581e2460 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.751+0000 7f4ba266e740 1 -- 172.21.15.38:0/1726534237 >> 172.21.15.38:0/1726534237 conn(0x556f581dd5b0 msgr2=0x556f581dd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 -- 172.21.15.38:0/1726534237 shutdown_connections 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 -- 172.21.15.38:0/1726534237 wait complete. 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 -- start start 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 Processor -- start 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581ec220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581ec760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556f581eab50 con 0x556f581e2a30 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556f58102f10 con 0x556f581e7ba0 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556f581eacd0 con 0x556f581e2040 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4ba266e740 10 client.0 _ll_register_callbacks cb 0x556f57e5abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.752+0000 7f4b9d01f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581ec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581ec220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581ec220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42200/0 (socket says 172.21.15.38:42200) 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9d820640 1 -- 172.21.15.38:0/4118723776 learned_addr learned my addr 172.21.15.38:0/4118723776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9d01f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581ec760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33026/0 (socket says 172.21.15.38:33026) 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 -- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 msgr2=0x556f581ec220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581ec220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 -- 172.21.15.38:0/4118723776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 msgr2=0x556f581ec760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581ec760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 -- 172.21.15.38:0/4118723776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f580d4c80 con 0x556f581e7ba0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9d820640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581ec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b9e021640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581f37a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b88037040 tx=0x7f4b88004460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1506c0fe88b65c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/4118723776 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88034440 con 0x556f581e7ba0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/4118723776 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4b88004760 con 0x556f581e7ba0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b88003e50 con 0x556f581e7ba0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/4118723776 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b88061050 con 0x556f581e7ba0 2024-04-22T01:15:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4ba266e740 10 client.25835 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.753+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556f57e5b3f0 con 0x556f581e7ba0 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/4118723776 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b88003e50 con 0x556f581e7ba0 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/4118723776 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4b8803f070 con 0x556f581e7ba0 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4ba266e740 10 client.25835 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4ba266e740 10 client.25835 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4ba266e740 20 client.25835 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4ba266e740 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f581f7c90 0x556f581fa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4ba266e740 4 client.25835 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f58154a70 con 0x556f581f7c90 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4b9d820640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f581f7c90 0x556f581fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4b9d820640 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f581f7c90 0x556f581fa080 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb5a52e1e0896f4a server_cookie=84055b2be735f009 in_seq=0 out_seq=0 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.754+0000 7f4b7d7fa640 10 client.25835 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/4118723776 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f58154a70 con 0x556f581f7c90 2024-04-22T01:15:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4b7d7fa640 10 client.25835 handle_command_reply: tid=0 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 client.25835 shutdown 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 msgr2=0x556f581f37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581f37a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b88037040 tx=0x7f4b88004460 comp rx=0 tx=0).stop 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 shutdown_connections 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f581f7c90 0x556f581fa080 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581f37a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581ec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 --2- 172.21.15.38:0/4118723776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4b7d7fa640 0 client.25835 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 >> 172.21.15.38:0/4118723776 conn(0x556f581dd5b0 msgr2=0x556f581dc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 shutdown_connections 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.755+0000 7f4ba266e740 1 -- 172.21.15.38:0/4118723776 wait complete. 2024-04-22T01:15:20.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.756+0000 7f4ba266e740 20 client.25835 trim_cache size 0 max 16384 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 Processor -- start 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 -- start start 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581dc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581f3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581e3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981ac700 con 0x556f581e2a30 2024-04-22T01:15:20.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981ae240 con 0x556f581e7ba0 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4ba266e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981acdd0 con 0x556f581e2040 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4b9d01f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4b9d01f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33028/0 (socket says 172.21.15.38:33028) 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581dc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.769+0000 7f4b9d01f640 1 -- 172.21.15.38:0/2712684021 learned_addr learned my addr 172.21.15.38:0/2712684021 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9e021640 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581e3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9d820640 1 -- 172.21.15.38:0/2712684021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 msgr2=0x556f581e3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9d820640 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581e3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9d820640 1 -- 172.21.15.38:0/2712684021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 msgr2=0x556f581f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9d820640 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9d820640 1 -- 172.21.15.38:0/2712684021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f57f257d0 con 0x556f581e2040 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b9d820640 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581dc6d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b90097cb0 tx=0x7f4b900cc450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa1c518fc9f01650 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b7f7fe640 1 -- 172.21.15.38:0/2712684021 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90020040 con 0x556f581e2040 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b7f7fe640 1 -- 172.21.15.38:0/2712684021 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b90095e50 con 0x556f581e2040 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.770+0000 7f4b7f7fe640 1 -- 172.21.15.38:0/2712684021 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b900a3f50 con 0x556f581e2040 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 -- 172.21.15.38:0/2712684021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 msgr2=0x556f581dc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581dc6d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b90097cb0 tx=0x7f4b900cc450 comp rx=0 tx=0).stop 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 -- 172.21.15.38:0/2712684021 shutdown_connections 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e7ba0 0x556f581e3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2a30 0x556f581f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 --2- 172.21.15.38:0/2712684021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e2040 0x556f581dc6d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 -- 172.21.15.38:0/2712684021 >> 172.21.15.38:0/2712684021 conn(0x556f581dd5b0 msgr2=0x556f581dbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 -- 172.21.15.38:0/2712684021 shutdown_connections 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 -- 172.21.15.38:0/2712684021 wait complete. 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 -- start start 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.771+0000 7f4ba266e740 1 Processor -- start 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 0x556f581f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2a30 0x556f581f0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e7ba0 0x556f581f0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981adb70 con 0x556f581e2040 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981ad4a0 con 0x556f581e2a30 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981aa6c0 con 0x556f581e7ba0 2024-04-22T01:15:20.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 10 client.0 _ll_register_callbacks cb 0x7f4b8c0095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 0x556f581f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 0x556f581f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33032/0 (socket says 172.21.15.38:33032) 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 -- 172.21.15.38:0/836154407 learned_addr learned my addr 172.21.15.38:0/836154407 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2a30 0x556f581f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9e021640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e7ba0 0x556f581f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 -- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e7ba0 msgr2=0x556f581f0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e7ba0 0x556f581f0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 -- 172.21.15.38:0/836154407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2a30 msgr2=0x556f581f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2a30 0x556f581f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 -- 172.21.15.38:0/836154407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f5825c0f0 con 0x556f581e2040 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b9d820640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 0x556f581f36c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4b9009afe0 tx=0x7f4b90003870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddda2744c345c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/836154407 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90020050 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/836154407 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4b900042a0 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b90095e50 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.772+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/836154407 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b9009b9e0 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 10 client.16356 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556f5825d3d0 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/836154407 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b90095e50 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/836154407 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4b9009f170 con 0x556f581e2040 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 10 client.16356 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 10 client.16356 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 20 client.16356 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f5825dd20 0x556f58260140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 4 client.16356 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f5814edc0 con 0x556f5825dd20 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f5825dd20 0x556f58260140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4b7d7fa640 10 client.16356 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.773+0000 7f4b9d01f640 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f5825dd20 0x556f58260140 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ffd81d948b9fa4a4 server_cookie=153c06a15abee69e in_seq=0 out_seq=0 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b7d7fa640 1 -- 172.21.15.38:0/836154407 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f5814edc0 con 0x556f5825dd20 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b7d7fa640 10 client.16356 handle_command_reply: tid=0 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 client.16356 shutdown 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 msgr2=0x556f581f36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 0x556f581f36c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4b9009afe0 tx=0x7f4b90003870 comp rx=0 tx=0).stop 2024-04-22T01:15:20.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 shutdown_connections 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x556f5825dd20 0x556f58260140 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556f581e7ba0 0x556f581f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556f581e2a30 0x556f581f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b7d7fa640 0 client.16356 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 --2- 172.21.15.38:0/836154407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556f581e2040 0x556f581f36c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 >> 172.21.15.38:0/836154407 conn(0x556f581dd5b0 msgr2=0x556f581e10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 shutdown_connections 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 1 -- 172.21.15.38:0/836154407 wait complete. 2024-04-22T01:15:20.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4ba266e740 20 client.16356 trim_cache size 0 max 16384 2024-04-22T01:15:20.783 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:20.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4b800a2b90 msgr2=0x7f4b800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4b800a2b90 0x7f4b800a5050 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4b8804b000 tx=0x7f4b88044000 comp rx=0 tx=0).stop 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 msgr2=0x7f4b980754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.774+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 0x7f4b980754f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000d770 tx=0x7f4b9000dc30 comp rx=0 tx=0).stop 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 shutdown_connections 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4b800a2b90 0x7f4b800a5050 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98103350 0x7f4b98074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b98102990 0x7f4b98073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 --2- 172.21.15.38:0/288509756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981017a0 0x7f4b980754f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 >> 172.21.15.38:0/288509756 conn(0x7f4b980fcd10 msgr2=0x7f4b98105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 shutdown_connections 2024-04-22T01:15:20.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.775+0000 7f4b9fa84640 1 -- 172.21.15.38:0/288509756 wait complete. 2024-04-22T01:15:20.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:20 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3403179184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:15:20.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- 172.21.15.38:0/1577781908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 msgr2=0x7f4880101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- 172.21.15.38:0/1577781908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880101cf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f486c009900 tx=0x7f486c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- 172.21.15.38:0/1577781908 shutdown_connections 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- 172.21.15.38:0/1577781908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880102230 0x7f4880108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- 172.21.15.38:0/1577781908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880101cf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- 172.21.15.38:0/1577781908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4880100680 0x7f4880100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- 172.21.15.38:0/1577781908 >> 172.21.15.38:0/1577781908 conn(0x7f48800fbbf0 msgr2=0x7f48800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- 172.21.15.38:0/1577781908 shutdown_connections 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- 172.21.15.38:0/1577781908 wait complete. 2024-04-22T01:15:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 Processor -- start 2024-04-22T01:15:20.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- start start 2024-04-22T01:15:20.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880100680 0x7f4880196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4880102230 0x7f488019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f488010c360 con 0x7f4880101870 2024-04-22T01:15:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f488010c1e0 con 0x7f4880102230 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f488707d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f488010c4e0 con 0x7f4880100680 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f487ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f487ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33062/0 (socket says 172.21.15.38:33062) 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.892+0000 7f487ffff640 1 -- 172.21.15.38:0/2415844126 learned_addr learned my addr 172.21.15.38:0/2415844126 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f488561a640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4880102230 0x7f488019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f4884e19640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880100680 0x7f4880196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487ffff640 1 -- 172.21.15.38:0/2415844126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880100680 msgr2=0x7f4880196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487ffff640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880100680 0x7f4880196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487ffff640 1 -- 172.21.15.38:0/2415844126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4880102230 msgr2=0x7f488019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487ffff640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4880102230 0x7f488019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487ffff640 1 -- 172.21.15.38:0/2415844126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488019e650 con 0x7f4880101870 2024-04-22T01:15:20.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f4884e19640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880100680 0x7f4880196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:20.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487ffff640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880196f70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f486c009230 tx=0x7f486c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487dffb640 1 -- 172.21.15.38:0/2415844126 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c041070 con 0x7f4880101870 2024-04-22T01:15:20.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488019e8e0 con 0x7f4880101870 2024-04-22T01:15:20.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487dffb640 1 -- 172.21.15.38:0/2415844126 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f486c0044d0 con 0x7f4880101870 2024-04-22T01:15:20.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f487dffb640 1 -- 172.21.15.38:0/2415844126 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c034440 con 0x7f4880101870 2024-04-22T01:15:20.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.893+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488019ee20 con 0x7f4880101870 2024-04-22T01:15:20.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.895+0000 7f487dffb640 1 -- 172.21.15.38:0/2415844126 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f486c044070 con 0x7f4880101870 2024-04-22T01:15:20.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 Processor -- start 2024-04-22T01:15:20.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 -- start start 2024-04-22T01:15:20.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7250 0x55fcc5af7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7c40 0x55fcc5afc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5aff240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fcc5a5a530 con 0x55fcc5afcdb0 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fcc5a02830 con 0x55fcc5af7c40 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4889c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fcc5a564f0 con 0x55fcc5af7250 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5aff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5aff240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33068/0 (socket says 172.21.15.38:33068) 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 -- 172.21.15.38:0/3052609691 learned_addr learned my addr 172.21.15.38:0/3052609691 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487dffb640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48700a2b90 0x7f48700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f4884e19640 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7c40 0x55fcc5afc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f488561a640 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7250 0x55fcc5af7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f488561a640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48700a2b90 0x7f48700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487dffb640 1 -- 172.21.15.38:0/2415844126 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f486c0f1110 con 0x7f4880101870 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 -- 172.21.15.38:0/3052609691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7250 msgr2=0x55fcc5af7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7250 0x55fcc5af7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 -- 172.21.15.38:0/3052609691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7c40 msgr2=0x55fcc5afc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7c40 0x55fcc5afc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.896+0000 7f487ffff640 1 -- 172.21.15.38:0/3052609691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcc5a262c0 con 0x55fcc5afcdb0 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f4884e19640 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7c40 0x55fcc5afc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f487ffff640 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5aff240 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f486c0b9a00 tx=0x7f486c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c52591a539c8b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f488561a640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48700a2b90 0x7f48700a5050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4878009900 tx=0x7f4878009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f486affd640 1 -- 172.21.15.38:0/3052609691 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c04b070 con 0x55fcc5afcdb0 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f486affd640 1 -- 172.21.15.38:0/3052609691 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f486c0bdca0 con 0x55fcc5afcdb0 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f486affd640 1 -- 172.21.15.38:0/3052609691 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c0c6210 con 0x55fcc5afcdb0 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.897+0000 7f4889c63740 1 -- 172.21.15.38:0/3052609691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 msgr2=0x55fcc5aff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5aff240 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f486c0b9a00 tx=0x7f486c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- 172.21.15.38:0/3052609691 shutdown_connections 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5aff240 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7c40 0x55fcc5afc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- 172.21.15.38:0/3052609691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7250 0x55fcc5af7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- 172.21.15.38:0/3052609691 >> 172.21.15.38:0/3052609691 conn(0x55fcc5af05b0 msgr2=0x55fcc5af09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- 172.21.15.38:0/3052609691 shutdown_connections 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- 172.21.15.38:0/3052609691 wait complete. 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- start start 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 Processor -- start 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b010c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 0x55fcc5b01600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b08640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fcc5a5a530 con 0x55fcc5af7c40 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fcc5a15f10 con 0x55fcc5af7250 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fcc5affb20 con 0x55fcc5afcdb0 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.898+0000 7f4889c63740 10 client.0 _ll_register_callbacks cb 0x55fcc576dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 0x55fcc5b01600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 0x55fcc5b01600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33082/0 (socket says 172.21.15.38:33082) 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 -- 172.21.15.38:0/3952694010 learned_addr learned my addr 172.21.15.38:0/3952694010 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f488561a640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b08640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f4884e19640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 -- 172.21.15.38:0/3952694010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5afcdb0 msgr2=0x55fcc5b08640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b08640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 -- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 msgr2=0x55fcc5b010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 -- 172.21.15.38:0/3952694010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcc59e7c80 con 0x55fcc5af7c40 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f487ffff640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 0x55fcc5b01600 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f486c0f10e0 tx=0x7f486c0c1ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bccfdbd97198a83c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f485ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f485ffff640 1 -- 172.21.15.38:0/3952694010 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c04b040 con 0x55fcc5af7c40 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f486c0bdca0 con 0x55fcc5af7c40 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f485ffff640 1 -- 172.21.15.38:0/3952694010 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f486c0b92f0 con 0x55fcc5af7c40 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f485ffff640 1 -- 172.21.15.38:0/3952694010 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f486c0bd370 con 0x55fcc5af7c40 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f4889c63740 10 client.16368 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.899+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcc576e3f0 con 0x55fcc5af7c40 2024-04-22T01:15:20.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f485ffff640 1 -- 172.21.15.38:0/3952694010 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f486c0bdca0 con 0x55fcc5af7c40 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f485ffff640 1 -- 172.21.15.38:0/3952694010 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f486c0c5da0 con 0x55fcc5af7c40 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4889c63740 10 client.16368 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4889c63740 10 client.16368 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4889c63740 20 client.16368 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4889c63740 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b0ccb0 0x55fcc5b0f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4889c63740 4 client.16368 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcc5a67a70 con 0x55fcc5b0ccb0 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4884e19640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b0ccb0 0x55fcc5b0f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f485ffff640 10 client.16368 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.900+0000 7f4884e19640 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b0ccb0 0x55fcc5b0f0a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ebd2e10b9421b3db server_cookie=5afccdac2a541d4b in_seq=0 out_seq=0 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f485ffff640 1 -- 172.21.15.38:0/3952694010 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcc5a67a70 con 0x55fcc5b0ccb0 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f485ffff640 10 client.16368 handle_command_reply: tid=0 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 client.16368 shutdown 2024-04-22T01:15:20.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 msgr2=0x55fcc5b01600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 0x55fcc5b01600 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f486c0f10e0 tx=0x7f486c0c1ab0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 shutdown_connections 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b0ccb0 0x55fcc5b0f0a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b08640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f485ffff640 0 client.16368 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7c40 0x55fcc5b01600 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 --2- 172.21.15.38:0/3952694010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 >> 172.21.15.38:0/3952694010 conn(0x55fcc5af05b0 msgr2=0x55fcc5aef850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 shutdown_connections 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.901+0000 7f4889c63740 1 -- 172.21.15.38:0/3952694010 wait complete. 2024-04-22T01:15:20.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.902+0000 7f4889c63740 20 client.16368 trim_cache size 0 max 16384 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 Processor -- start 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 -- start start 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 0x55fcc5af3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b08600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5afcdb0 0x55fcc5af86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48801a83f0 con 0x55fcc5af7250 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48801a9f30 con 0x55fcc5afcdb0 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4889c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48801a8ac0 con 0x55fcc5af7c40 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 0x55fcc5af3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 0x55fcc5af3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33094/0 (socket says 172.21.15.38:33094) 2024-04-22T01:15:20.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 -- 172.21.15.38:0/4025979379 learned_addr learned my addr 172.21.15.38:0/4025979379 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f487ffff640 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b08600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 -- 172.21.15.38:0/4025979379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 msgr2=0x55fcc5b08600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b08600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 -- 172.21.15.38:0/4025979379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5afcdb0 msgr2=0x55fcc5af86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5afcdb0 0x55fcc5af86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f4884e19640 1 -- 172.21.15.38:0/4025979379 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcc58387d0 con 0x55fcc5af7250 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.915+0000 7f487ffff640 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b08600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4884e19640 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 0x55fcc5af3f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4874000c00 tx=0x7f4874009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66711176f8a0a2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f486a7fc640 1 -- 172.21.15.38:0/4025979379 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487401e430 con 0x55fcc5af7250 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f486a7fc640 1 -- 172.21.15.38:0/4025979379 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4874005020 con 0x55fcc5af7250 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f486a7fc640 1 -- 172.21.15.38:0/4025979379 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874004a90 con 0x55fcc5af7250 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 -- 172.21.15.38:0/4025979379 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 msgr2=0x55fcc5af3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 0x55fcc5af3f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4874000c00 tx=0x7f4874009420 comp rx=0 tx=0).stop 2024-04-22T01:15:20.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 -- 172.21.15.38:0/4025979379 shutdown_connections 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5afcdb0 0x55fcc5af86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b08600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 --2- 172.21.15.38:0/4025979379 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5af7250 0x55fcc5af3f20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.916+0000 7f4889c63740 1 -- 172.21.15.38:0/4025979379 >> 172.21.15.38:0/4025979379 conn(0x55fcc5af05b0 msgr2=0x55fcc5af3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 -- 172.21.15.38:0/4025979379 shutdown_connections 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 -- 172.21.15.38:0/4025979379 wait complete. 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 -- start start 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 Processor -- start 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b05530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b05a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b71210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48801a9860 con 0x55fcc5afcdb0 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f48801a9190 con 0x55fcc5af7250 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48801a63b0 con 0x55fcc5af7c40 2024-04-22T01:15:20.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f4889c63740 10 client.0 _ll_register_callbacks cb 0x7f4874009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f488561a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b71210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f488561a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b71210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33108/0 (socket says 172.21.15.38:33108) 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.917+0000 7f488561a640 1 -- 172.21.15.38:0/223155359 learned_addr learned my addr 172.21.15.38:0/223155359 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f487ffff640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b05a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f4884e19640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b05530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f488561a640 1 -- 172.21.15.38:0/223155359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 msgr2=0x55fcc5b05a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f488561a640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b05a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f488561a640 1 -- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 msgr2=0x55fcc5b05530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f488561a640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b05530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f488561a640 1 -- 172.21.15.38:0/223155359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcc5b71750 con 0x55fcc5afcdb0 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f4884e19640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b05530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f488561a640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b71210 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4878019410 tx=0x7f4878009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9b5ce30374680f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f485ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f485ffff640 1 -- 172.21.15.38:0/223155359 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878014040 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f485ffff640 1 -- 172.21.15.38:0/223155359 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f4878025460 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4874005020 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f485ffff640 1 -- 172.21.15.38:0/223155359 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4878033050 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f4889c63740 10 client.16374 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcc5b72ad0 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.918+0000 7f485ffff640 1 -- 172.21.15.38:0/223155359 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4874005020 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f485ffff640 1 -- 172.21.15.38:0/223155359 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4878024760 con 0x55fcc5afcdb0 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 10 client.16374 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 10 client.16374 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 20 client.16374 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b73530 0x55fcc5b75950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 4 client.16374 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcc5a61dc0 con 0x55fcc5b73530 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4884e19640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b73530 0x55fcc5b75950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4884e19640 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b73530 0x55fcc5b75950 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=68d7bf12b52566a2 server_cookie=76f94176708e5209 in_seq=0 out_seq=0 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f485ffff640 10 client.16374 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f485ffff640 1 -- 172.21.15.38:0/223155359 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcc5a61dc0 con 0x55fcc5b73530 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f485ffff640 10 client.16374 handle_command_reply: tid=0 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 1 client.16374 shutdown 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 msgr2=0x55fcc5b71210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.919+0000 7f4889c63740 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b71210 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4878019410 tx=0x7f4878009a10 comp rx=0 tx=0).stop 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 shutdown_connections 2024-04-22T01:15:20.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55fcc5b73530 0x55fcc5b75950 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fcc5afcdb0 0x55fcc5b71210 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fcc5af7c40 0x55fcc5b05a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 --2- 172.21.15.38:0/223155359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcc5af7250 0x55fcc5b05530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 >> 172.21.15.38:0/223155359 conn(0x55fcc5af05b0 msgr2=0x55fcc5af40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f485ffff640 0 client.16374 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 shutdown_connections 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 1 -- 172.21.15.38:0/223155359 wait complete. 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f4889c63740 20 client.16374 trim_cache size 0 max 16384 2024-04-22T01:15:20.927 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48700a2b90 msgr2=0x7f48700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48700a2b90 0x7f48700a5050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4878009900 tx=0x7f4878009650 comp rx=0 tx=0).stop 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 msgr2=0x7f4880196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880196f70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f486c009230 tx=0x7f486c005d50 comp rx=0 tx=0).stop 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 shutdown_connections 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48700a2b90 0x7f48700a5050 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4880102230 0x7f488019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4880101870 0x7f4880196f70 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 --2- 172.21.15.38:0/2415844126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4880100680 0x7f4880196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:20.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 >> 172.21.15.38:0/2415844126 conn(0x7f48800fbbf0 msgr2=0x7f4880104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:20.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 shutdown_connections 2024-04-22T01:15:20.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:20.920+0000 7f488707d640 1 -- 172.21.15.38:0/2415844126 wait complete. 2024-04-22T01:15:20.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T01:15:21.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:20 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3403179184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:15:21.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 -- 172.21.15.38:0/1638441175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50341006a0 msgr2=0x7f5034100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 --2- 172.21.15.38:0/1638441175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50341006a0 0x7f5034100ac0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5030009900 tx=0x7f503002bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 -- 172.21.15.38:0/1638441175 shutdown_connections 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 --2- 172.21.15.38:0/1638441175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5034102250 0x7f5034108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 --2- 172.21.15.38:0/1638441175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 0x7f5034101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 --2- 172.21.15.38:0/1638441175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50341006a0 0x7f5034100ac0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 -- 172.21.15.38:0/1638441175 >> 172.21.15.38:0/1638441175 conn(0x7f50340fbc30 msgr2=0x7f50340fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.035+0000 7f503c765640 1 -- 172.21.15.38:0/1638441175 shutdown_connections 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 -- 172.21.15.38:0/1638441175 wait complete. 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 Processor -- start 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 -- start start 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50341006a0 0x7f503406a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 0x7f503419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 0x7f503419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f503410c320 con 0x7f5034102250 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f503410c1a0 con 0x7f5034101890 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503c765640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f503410c4a0 con 0x7f50341006a0 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503ad02640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 0x7f503419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503ad02640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 0x7f503419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33128/0 (socket says 172.21.15.38:33128) 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503ad02640 1 -- 172.21.15.38:0/906824750 learned_addr learned my addr 172.21.15.38:0/906824750 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.036+0000 7f503a501640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50341006a0 0x7f503406a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f5039d00640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 0x7f503419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503ad02640 1 -- 172.21.15.38:0/906824750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50341006a0 msgr2=0x7f503406a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503ad02640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50341006a0 0x7f503406a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503ad02640 1 -- 172.21.15.38:0/906824750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 msgr2=0x7f503419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503ad02640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 0x7f503419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503ad02640 1 -- 172.21.15.38:0/906824750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503419f730 con 0x7f5034102250 2024-04-22T01:15:21.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f5039d00640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 0x7f503419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:21.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503ad02640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 0x7f503419ee20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f502c00eaa0 tx=0x7f502c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f502b7fe640 1 -- 172.21.15.38:0/906824750 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c00cdb0 con 0x7f5034102250 2024-04-22T01:15:21.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f502b7fe640 1 -- 172.21.15.38:0/906824750 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f502c005020 con 0x7f5034102250 2024-04-22T01:15:21.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503419f9c0 con 0x7f5034102250 2024-04-22T01:15:21.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f502b7fe640 1 -- 172.21.15.38:0/906824750 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c0116b0 con 0x7f5034102250 2024-04-22T01:15:21.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.037+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f503419ff10 con 0x7f5034102250 2024-04-22T01:15:21.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.039+0000 7f502b7fe640 1 -- 172.21.15.38:0/906824750 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f502c005560 con 0x7f5034102250 2024-04-22T01:15:21.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 Processor -- start 2024-04-22T01:15:21.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 -- start start 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332bf6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332bfb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bfbfb0 0x55e332bfe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e332b57530 con 0x55e332bf6e40 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e332aff830 con 0x55e332bfbfb0 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503f42c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e332b534f0 con 0x55e332bf6450 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f502b7fe640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50200a2b90 0x7f50200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332bfb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332bfb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33130/0 (socket says 172.21.15.38:33130) 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 -- 172.21.15.38:0/771753163 learned_addr learned my addr 172.21.15.38:0/771753163 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f5039d00640 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bfbfb0 0x55e332bfe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503ad02640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50200a2b90 0x7f50200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f5039d00640 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332bf6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f502b7fe640 1 -- 172.21.15.38:0/906824750 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f502c0cf640 con 0x7f5034102250 2024-04-22T01:15:21.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 -- 172.21.15.38:0/771753163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 msgr2=0x55e332bf6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332bf6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 -- 172.21.15.38:0/771753163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bfbfb0 msgr2=0x55e332bfe440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bfbfb0 0x55e332bfe440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f503a501640 1 -- 172.21.15.38:0/771753163 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e332b232c0 con 0x55e332bf6e40 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.040+0000 7f5039d00640 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bfbfb0 0x55e332bfe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.041+0000 7f503a501640 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332bfb940 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5030009900 tx=0x7f50300095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae27cd66359f5150 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.041+0000 7f503ad02640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50200a2b90 0x7f50200a5050 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f502c0a4210 tx=0x7f502c0054d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.041+0000 7f5028ff9640 1 -- 172.21.15.38:0/771753163 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503002cb80 con 0x55e332bf6e40 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.041+0000 7f5028ff9640 1 -- 172.21.15.38:0/771753163 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5030005040 con 0x55e332bf6e40 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.041+0000 7f5028ff9640 1 -- 172.21.15.38:0/771753163 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030004920 con 0x55e332bf6e40 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- 172.21.15.38:0/771753163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 msgr2=0x55e332bfb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332bfb940 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5030009900 tx=0x7f50300095e0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- 172.21.15.38:0/771753163 shutdown_connections 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bfbfb0 0x55e332bfe440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332bfb940 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- 172.21.15.38:0/771753163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332bf6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- 172.21.15.38:0/771753163 >> 172.21.15.38:0/771753163 conn(0x55e332bed5b0 msgr2=0x55e332bed9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- 172.21.15.38:0/771753163 shutdown_connections 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- 172.21.15.38:0/771753163 wait complete. 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- start start 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 Processor -- start 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332c00630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332c00b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bfbfb0 0x55e332c07bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e332bfef00 con 0x55e332bf6e40 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e332b12f10 con 0x55e332bf6450 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e332bff080 con 0x55e332bfbfb0 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.042+0000 7f503f42c740 10 client.0 _ll_register_callbacks cb 0x55e33286abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332c00b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332c00b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33132/0 (socket says 172.21.15.38:33132) 2024-04-22T01:15:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 -- 172.21.15.38:0/2484364149 learned_addr learned my addr 172.21.15.38:0/2484364149 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f5039d00640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bfbfb0 0x55e332c07bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f5039d00640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332c00630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 -- 172.21.15.38:0/2484364149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bfbfb0 msgr2=0x55e332c07bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bfbfb0 0x55e332c07bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 -- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 msgr2=0x55e332c00630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332c00630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 -- 172.21.15.38:0/2484364149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e332ae4c80 con 0x55e332bf6e40 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503a501640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332c00b70 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f503002c170 tx=0x7f5030005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2e7fdd30e0d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f501dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f501dffb640 1 -- 172.21.15.38:0/2484364149 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030004ad0 con 0x55e332bf6e40 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f501dffb640 1 -- 172.21.15.38:0/2484364149 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f5030004c70 con 0x55e332bf6e40 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5030005040 con 0x55e332bf6e40 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f501dffb640 1 -- 172.21.15.38:0/2484364149 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5030004e70 con 0x55e332bf6e40 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503f42c740 10 client.16386 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:21.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.043+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e33286b3f0 con 0x55e332bf6e40 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f501dffb640 1 -- 172.21.15.38:0/2484364149 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5030005040 con 0x55e332bf6e40 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f501dffb640 1 -- 172.21.15.38:0/2484364149 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f503004e430 con 0x55e332bf6e40 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f503f42c740 10 client.16386 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f503f42c740 10 client.16386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f503f42c740 20 client.16386 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f503f42c740 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c0c0a0 0x55e332c0e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f503f42c740 4 client.16386 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e332b64a70 con 0x55e332c0c0a0 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f5039d00640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c0c0a0 0x55e332c0e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f501dffb640 10 client.16386 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.044+0000 7f5039d00640 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c0c0a0 0x55e332c0e490 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed0d5812d9f0f3c9 server_cookie=2afcd2a04cdd71a4 in_seq=0 out_seq=0 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f501dffb640 1 -- 172.21.15.38:0/2484364149 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e332b64a70 con 0x55e332c0c0a0 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f501dffb640 10 client.16386 handle_command_reply: tid=0 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 client.16386 shutdown 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 msgr2=0x55e332c00b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332c00b70 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f503002c170 tx=0x7f5030005420 comp rx=0 tx=0).stop 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 shutdown_connections 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c0c0a0 0x55e332c0e490 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bfbfb0 0x55e332c07bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bf6e40 0x55e332c00b70 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f501dffb640 0 client.16386 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 --2- 172.21.15.38:0/2484364149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332c00630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 >> 172.21.15.38:0/2484364149 conn(0x55e332bed5b0 msgr2=0x55e332bec850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 shutdown_connections 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 1 -- 172.21.15.38:0/2484364149 wait complete. 2024-04-22T01:15:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.045+0000 7f503f42c740 20 client.16386 trim_cache size 0 max 16384 2024-04-22T01:15:21.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 Processor -- start 2024-04-22T01:15:21.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 -- start start 2024-04-22T01:15:21.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332bffc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6e40 0x55e332c07b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332bf76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f50341a82c0 con 0x55e332bfbfb0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50341a9e00 con 0x55e332bf6450 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503f42c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50341a8990 con 0x55e332bf6e40 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332bf76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332bf76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33136/0 (socket says 172.21.15.38:33136) 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 -- 172.21.15.38:0/4269452122 learned_addr learned my addr 172.21.15.38:0/4269452122 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332bffc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f503a501640 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6e40 0x55e332c07b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 -- 172.21.15.38:0/4269452122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6e40 msgr2=0x55e332c07b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6e40 0x55e332c07b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 -- 172.21.15.38:0/4269452122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 msgr2=0x55e332bffc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332bffc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 -- 172.21.15.38:0/4269452122 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3329357d0 con 0x55e332bfbfb0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.059+0000 7f5039d00640 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332bf76d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5024014900 tx=0x7f50240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9becf23f52aabd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f501ffff640 1 -- 172.21.15.38:0/4269452122 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5024011ca0 con 0x55e332bfbfb0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f501ffff640 1 -- 172.21.15.38:0/4269452122 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5024011e40 con 0x55e332bfbfb0 2024-04-22T01:15:21.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f501ffff640 1 -- 172.21.15.38:0/4269452122 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50240228a0 con 0x55e332bfbfb0 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 -- 172.21.15.38:0/4269452122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 msgr2=0x55e332bf76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332bf76d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5024014900 tx=0x7f50240046c0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 -- 172.21.15.38:0/4269452122 shutdown_connections 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332bf76d0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6e40 0x55e332c07b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 --2- 172.21.15.38:0/4269452122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6450 0x55e332bffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 -- 172.21.15.38:0/4269452122 >> 172.21.15.38:0/4269452122 conn(0x55e332bed5b0 msgr2=0x55e332bf10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 -- 172.21.15.38:0/4269452122 shutdown_connections 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 -- 172.21.15.38:0/4269452122 wait complete. 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.060+0000 7f503f42c740 1 -- start start 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 Processor -- start 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332c05490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6e40 0x55e332c059d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332c069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f50341a9730 con 0x55e332bfbfb0 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f50341a9060 con 0x55e332bf6e40 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50341a6280 con 0x55e332bf6450 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503f42c740 10 client.0 _ll_register_callbacks cb 0x7f5024009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332c069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332c069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33146/0 (socket says 172.21.15.38:33146) 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 -- 172.21.15.38:0/3376044023 learned_addr learned my addr 172.21.15.38:0/3376044023 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332c05490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f503a501640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6e40 0x55e332c059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 -- 172.21.15.38:0/3376044023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 msgr2=0x55e332c05490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332c05490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 -- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6e40 msgr2=0x55e332c059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6e40 0x55e332c059d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 -- 172.21.15.38:0/3376044023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e332c06fd0 con 0x55e332bfbfb0 2024-04-22T01:15:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f5039d00640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332c069e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5024014c90 tx=0x7f5024005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83000ed6e5d2afc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f501dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.061+0000 7f501dffb640 1 -- 172.21.15.38:0/3376044023 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5024022820 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f501dffb640 1 -- 172.21.15.38:0/3376044023 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f5024005050 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5024011e40 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f501dffb640 1 -- 172.21.15.38:0/3376044023 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5024005250 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 10 client.16392 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e332c725a0 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f501dffb640 1 -- 172.21.15.38:0/3376044023 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5024011e40 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f501dffb640 1 -- 172.21.15.38:0/3376044023 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f502402e040 con 0x55e332bfbfb0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 10 client.16392 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 20 client.16392 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c72ee0 0x55e332c75300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 4 client.16392 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e332c00420 con 0x55e332c72ee0 2024-04-22T01:15:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.062+0000 7f503a501640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c72ee0 0x55e332c75300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503a501640 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c72ee0 0x55e332c75300 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d14272b9fdedbee7 server_cookie=df6503ecc438df25 in_seq=0 out_seq=0 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f501dffb640 10 client.16392 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f501dffb640 1 -- 172.21.15.38:0/3376044023 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e332c00420 con 0x55e332c72ee0 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f501dffb640 10 client.16392 handle_command_reply: tid=0 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 client.16392 shutdown 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 msgr2=0x55e332c069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332c069e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5024014c90 tx=0x7f5024005850 comp rx=0 tx=0).stop 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 shutdown_connections 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e332c72ee0 0x55e332c75300 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e332bfbfb0 0x55e332c069e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e332bf6e40 0x55e332c059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f501dffb640 0 client.16392 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 --2- 172.21.15.38:0/3376044023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e332bf6450 0x55e332c05490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 >> 172.21.15.38:0/3376044023 conn(0x55e332bed5b0 msgr2=0x55e332bf10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 shutdown_connections 2024-04-22T01:15:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 1 -- 172.21.15.38:0/3376044023 wait complete. 2024-04-22T01:15:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503f42c740 20 client.16392 trim_cache size 0 max 16384 2024-04-22T01:15:21.070 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50200a2b90 msgr2=0x7f50200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503c765640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50200a2b90 0x7f50200a5050 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f502c0a4210 tx=0x7f502c0054d0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 msgr2=0x7f503419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.063+0000 7f503c765640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 0x7f503419ee20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f502c00eaa0 tx=0x7f502c00ef60 comp rx=0 tx=0).stop 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 shutdown_connections 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f50200a2b90 0x7f50200a5050 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5034102250 0x7f503419ee20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5034101890 0x7f503419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 --2- 172.21.15.38:0/906824750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50341006a0 0x7f503406a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 >> 172.21.15.38:0/906824750 conn(0x7f50340fbc30 msgr2=0x7f5034104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 shutdown_connections 2024-04-22T01:15:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:21.064+0000 7f503c765640 1 -- 172.21.15.38:0/906824750 wait complete. 2024-04-22T01:15:21.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:20 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3403179184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.604+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1099860731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc102a30 msgr2=0x7fa9bc104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.604+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1099860731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc102a30 0x7fa9bc104ea0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009900 tx=0x7fa9ac02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.604+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1099860731 shutdown_connections 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.604+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1099860731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9bc1053e0 0x7fa9bc105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.604+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1099860731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc102a30 0x7fa9bc104ea0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.604+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1099860731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1099860731 >> 172.21.15.38:0/1099860731 conn(0x7fa9bc0fbcc0 msgr2=0x7fa9bc0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1099860731 shutdown_connections 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1099860731 wait complete. 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 Processor -- start 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- start start 2024-04-22T01:15:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9bc102a30 0x7fa9bc19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9bc1053e0 0x7fa9bc1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc113a50 con 0x7fa9bc1000c0 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1138d0 con 0x7fa9bc102a30 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.605+0000 7fa9c3ec6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc113bd0 con 0x7fa9bc1053e0 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc19e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33172/0 (socket says 172.21.15.38:33172) 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 -- 172.21.15.38:0/1086002092 learned_addr learned my addr 172.21.15.38:0/1086002092 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1461640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9bc102a30 0x7fa9bc19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c2463640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9bc1053e0 0x7fa9bc1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 -- 172.21.15.38:0/1086002092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9bc1053e0 msgr2=0x7fa9bc1a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9bc1053e0 0x7fa9bc1a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 -- 172.21.15.38:0/1086002092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9bc102a30 msgr2=0x7fa9bc19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9bc102a30 0x7fa9bc19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 -- 172.21.15.38:0/1086002092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc1a5bd0 con 0x7fa9bc1000c0 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc19e000 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800e9f0 tx=0x7fa9b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9b2ffd640 1 -- 172.21.15.38:0/1086002092 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b800cd80 con 0x7fa9bc1000c0 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9b2ffd640 1 -- 172.21.15.38:0/1086002092 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9b8005500 con 0x7fa9bc1000c0 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc1a5ec0 con 0x7fa9bc1000c0 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9b2ffd640 1 -- 172.21.15.38:0/1086002092 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8010690 con 0x7fa9bc1000c0 2024-04-22T01:15:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.606+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc1a62d0 con 0x7fa9bc1000c0 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.608+0000 7fa9b2ffd640 1 -- 172.21.15.38:0/1086002092 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa9b80046c0 con 0x7fa9bc1000c0 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 Processor -- start 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 -- start start 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db2d30 0x561175db3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 0x561175dbad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561175d17b70 con 0x561175db3720 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561175cc0470 con 0x561175db8890 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.609+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561175d13b30 con 0x561175db2d30 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1c62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 0x561175dbad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1c62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33184/0 (socket says 172.21.15.38:33184) 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 0x561175dbad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55754/0 (socket says 172.21.15.38:55754) 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1c62640 1 -- 172.21.15.38:0/1993111765 learned_addr learned my addr 172.21.15.38:0/1993111765 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c2463640 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db2d30 0x561175db3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9b2ffd640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa9a80a2c30 0x7fa9a80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c2463640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa9a80a2c30 0x7fa9a80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 -- 172.21.15.38:0/1993111765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db2d30 msgr2=0x561175db3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db2d30 0x561175db3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 -- 172.21.15.38:0/1993111765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 msgr2=0x561175db8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 -- 172.21.15.38:0/1993111765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561175a2c210 con 0x561175db8890 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9b2ffd640 1 -- 172.21.15.38:0/1086002092 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa9b80cf920 con 0x7fa9bc1000c0 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c1461640 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 0x561175dbad20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009230 tx=0x7fa9ac0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=588431c9bbb223c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9c2463640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa9a80a2c30 0x7fa9a80a50f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x561175db3f80 tx=0x7fa9b4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9a7fff640 1 -- 172.21.15.38:0/1993111765 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac02cb80 con 0x561175db8890 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9a7fff640 1 -- 172.21.15.38:0/1993111765 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9ac005040 con 0x561175db8890 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.610+0000 7fa9a7fff640 1 -- 172.21.15.38:0/1993111765 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac004920 con 0x561175db8890 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 -- 172.21.15.38:0/1993111765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 msgr2=0x561175dbad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 0x561175dbad20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009230 tx=0x7fa9ac0095e0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 -- 172.21.15.38:0/1993111765 shutdown_connections 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db8890 0x561175dbad20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/1993111765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db2d30 0x561175db3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 -- 172.21.15.38:0/1993111765 >> 172.21.15.38:0/1993111765 conn(0x561175dae2a0 msgr2=0x561175dae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 -- 172.21.15.38:0/1993111765 shutdown_connections 2024-04-22T01:15:21.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.611+0000 7fa9c6b94740 1 -- 172.21.15.38:0/1993111765 wait complete. 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 -- start start 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 Processor -- start 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175dbd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db3720 0x561175dbd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 0x561175dc45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561175dbb7e0 con 0x561175db8890 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561175cd3a10 con 0x561175db2d30 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561175dbb960 con 0x561175db3720 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c6b94740 10 client.0 _ll_register_callbacks cb 0x561175d0eb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c2463640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 0x561175dc45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c2463640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 0x561175dc45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33198/0 (socket says 172.21.15.38:33198) 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c2463640 1 -- 172.21.15.38:0/2307039739 learned_addr learned my addr 172.21.15.38:0/2307039739 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c1461640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db3720 0x561175dbd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.612+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175dbd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c2463640 1 -- 172.21.15.38:0/2307039739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db3720 msgr2=0x561175dbd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c2463640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db3720 0x561175dbd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c2463640 1 -- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 msgr2=0x561175dbd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c2463640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175dbd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c2463640 1 -- 172.21.15.38:0/2307039739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561175af8620 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c2463640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 0x561175dc45b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4012b10 tx=0x7fa9b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455b8b6464cbc4a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9a57fa640 1 -- 172.21.15.38:0/2307039739 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4024070 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9a57fa640 1 -- 172.21.15.38:0/2307039739 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa9b4029460 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9ac005040 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9a57fa640 1 -- 172.21.15.38:0/2307039739 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa9b401a070 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c6b94740 10 client.16404 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561175dc79f0 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9a57fa640 1 -- 172.21.15.38:0/2307039739 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9ac005040 con 0x561175db8890 2024-04-22T01:15:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.613+0000 7fa9a57fa640 1 -- 172.21.15.38:0/2307039739 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa9b400f040 con 0x561175db8890 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c6b94740 10 client.16404 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c6b94740 10 client.16404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c6b94740 20 client.16404 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175dc9110 0x561175dcb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c6b94740 4 client.16404 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561175d250b0 con 0x561175dc9110 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175dc9110 0x561175dcb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9a57fa640 10 client.16404 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.614+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175dc9110 0x561175dcb5a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84d2401d7b97f4d0 server_cookie=e7746de6fbe8b953 in_seq=0 out_seq=0 2024-04-22T01:15:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9a57fa640 1 -- 172.21.15.38:0/2307039739 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561175d250b0 con 0x561175dc9110 2024-04-22T01:15:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9a57fa640 10 client.16404 handle_command_reply: tid=0 2024-04-22T01:15:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 client.16404 shutdown 2024-04-22T01:15:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 msgr2=0x561175dc45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 0x561175dc45b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4012b10 tx=0x7fa9b4004a40 comp rx=0 tx=0).stop 2024-04-22T01:15:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 shutdown_connections 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175dc9110 0x561175dcb5a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db8890 0x561175dc45b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9a57fa640 0 client.16404 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db3720 0x561175dbd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/2307039739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175dbd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 >> 172.21.15.38:0/2307039739 conn(0x561175dae2a0 msgr2=0x561175dad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 shutdown_connections 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 1 -- 172.21.15.38:0/2307039739 wait complete. 2024-04-22T01:15:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.615+0000 7fa9c6b94740 20 client.16404 trim_cache size 0 max 16384 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 Processor -- start 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 -- start start 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175db4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175db1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b0960 con 0x561175db3720 2024-04-22T01:15:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b24a0 con 0x561175db2d30 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b1030 con 0x561175db8890 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c1461640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c1461640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33210/0 (socket says 172.21.15.38:33210) 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c1461640 1 -- 172.21.15.38:0/3849643010 learned_addr learned my addr 172.21.15.38:0/3849643010 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.630+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175db4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c2463640 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175db1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1461640 1 -- 172.21.15.38:0/3849643010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 msgr2=0x561175db1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1461640 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175db1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1461640 1 -- 172.21.15.38:0/3849643010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 msgr2=0x561175db4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1461640 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175db4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1461640 1 -- 172.21.15.38:0/3849643010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561175db3b00 con 0x561175db3720 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175db4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9c1461640 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db4fc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac000c00 tx=0x7fa9ac005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7569acaf041fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9a77fe640 1 -- 172.21.15.38:0/3849643010 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac04d070 con 0x561175db3720 2024-04-22T01:15:21.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9a77fe640 1 -- 172.21.15.38:0/3849643010 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9ac02cb80 con 0x561175db3720 2024-04-22T01:15:21.633 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.631+0000 7fa9a77fe640 1 -- 172.21.15.38:0/3849643010 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac004920 con 0x561175db3720 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3849643010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 msgr2=0x561175db4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db4fc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac000c00 tx=0x7fa9ac005e50 comp rx=0 tx=0).stop 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3849643010 shutdown_connections 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175db1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db3720 0x561175db4fc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3849643010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db2d30 0x561175db4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3849643010 >> 172.21.15.38:0/3849643010 conn(0x561175dae2a0 msgr2=0x561175dae680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3849643010 shutdown_connections 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3849643010 wait complete. 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- start start 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 Processor -- start 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 0x561175dc1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db3720 0x561175dc19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175dc2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b1dd0 con 0x561175db2d30 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b1700 con 0x561175db3720 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1ae920 con 0x561175db8890 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.632+0000 7fa9c6b94740 10 client.0 _ll_register_callbacks cb 0x7fa9b4009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 0x561175dc1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 0x561175dc1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33212/0 (socket says 172.21.15.38:33212) 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 -- 172.21.15.38:0/3522069793 learned_addr learned my addr 172.21.15.38:0/3522069793 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1461640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db3720 0x561175dc19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c2463640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175dc2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 -- 172.21.15.38:0/3522069793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 msgr2=0x561175dc2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175dc2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 -- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db3720 msgr2=0x561175dc19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db3720 0x561175dc19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 -- 172.21.15.38:0/3522069793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561175dc2f50 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c2463640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175dc2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c1c62640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 0x561175dc1460 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800c6a0 tx=0x7fa9b80d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b94eadd8d4d7d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9a57fa640 1 -- 172.21.15.38:0/3522069793 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80b8070 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9ac02cb80 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9a57fa640 1 -- 172.21.15.38:0/3522069793 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa9b80983e0 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9a57fa640 1 -- 172.21.15.38:0/3522069793 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa9b809c470 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c6b94740 10 client.16410 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.633+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561175e2fa00 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9a57fa640 1 -- 172.21.15.38:0/3522069793 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9ac02cb80 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9a57fa640 1 -- 172.21.15.38:0/3522069793 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa9b809c060 con 0x561175db2d30 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c6b94740 10 client.16410 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c6b94740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c6b94740 20 client.16410 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175e303d0 0x561175e327f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c6b94740 4 client.16410 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:15:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561175d1f400 con 0x561175e303d0 2024-04-22T01:15:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c1461640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175e303d0 0x561175e327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9a57fa640 10 client.16410 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.634+0000 7fa9c1461640 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175e303d0 0x561175e327f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e709c20ac72c7e3 server_cookie=5e7f06ea514366d0 in_seq=0 out_seq=0 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9a57fa640 1 -- 172.21.15.38:0/3522069793 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561175d1f400 con 0x561175e303d0 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9a57fa640 10 client.16410 handle_command_reply: tid=0 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 client.16410 shutdown 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 msgr2=0x561175dc1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 0x561175dc1460 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800c6a0 tx=0x7fa9b80d1040 comp rx=0 tx=0).stop 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 shutdown_connections 2024-04-22T01:15:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x561175e303d0 0x561175e327f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561175db8890 0x561175dc2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561175db3720 0x561175dc19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9a57fa640 0 client.16410 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 --2- 172.21.15.38:0/3522069793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561175db2d30 0x561175dc1460 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 >> 172.21.15.38:0/3522069793 conn(0x561175dae2a0 msgr2=0x561175dbcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 shutdown_connections 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 1 -- 172.21.15.38:0/3522069793 wait complete. 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c6b94740 20 client.16410 trim_cache size 0 max 16384 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:15:21.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa9a80a2c30 msgr2=0x7fa9a80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa9a80a2c30 0x7fa9a80a50f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x561175db3f80 tx=0x7fa9b4004ab0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 msgr2=0x7fa9bc19e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.646+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc19e000 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800e9f0 tx=0x7fa9b800edf0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 shutdown_connections 2024-04-22T01:15:21.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa9a80a2c30 0x7fa9a80a50f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9bc1053e0 0x7fa9bc1a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9bc102a30 0x7fa9bc19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 --2- 172.21.15.38:0/1086002092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9bc1000c0 0x7fa9bc19e000 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 >> 172.21.15.38:0/1086002092 conn(0x7fa9bc0fbcc0 msgr2=0x7fa9bc100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 shutdown_connections 2024-04-22T01:15:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:21.647+0000 7fa9c3ec6640 1 -- 172.21.15.38:0/1086002092 wait complete. 2024-04-22T01:15:22.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:21 smithi203 ceph-mon[31825]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 691 MiB/s wr, 310 op/s 2024-04-22T01:15:22.075 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T01:15:22.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:15:22.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T01:15:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:21 smithi038 ceph-mon[28043]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 691 MiB/s wr, 310 op/s 2024-04-22T01:15:22.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:21 smithi111 ceph-mon[31057]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 691 MiB/s wr, 310 op/s 2024-04-22T01:15:22.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 -- 172.21.15.38:0/3542201174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 msgr2=0x7fd66c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 --2- 172.21.15.38:0/3542201174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c100ac0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd668009900 tx=0x7fd66802bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:22.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 -- 172.21.15.38:0/3542201174 shutdown_connections 2024-04-22T01:15:22.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 --2- 172.21.15.38:0/3542201174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 0x7fd66c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 --2- 172.21.15.38:0/3542201174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd66c101890 0x7fd66c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 --2- 172.21.15.38:0/3542201174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c100ac0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 -- 172.21.15.38:0/3542201174 >> 172.21.15.38:0/3542201174 conn(0x7fd66c0fbc30 msgr2=0x7fd66c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.195+0000 7fd672d8a640 1 -- 172.21.15.38:0/3542201174 shutdown_connections 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 -- 172.21.15.38:0/3542201174 wait complete. 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 Processor -- start 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 -- start start 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c10b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd66c101890 0x7fd66c109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 0x7fd66c109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10d440 con 0x7fd66c1006a0 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10d2c0 con 0x7fd66c102250 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd672d8a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10d5c0 con 0x7fd66c101890 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd670b26640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c10b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd670b26640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c10b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33220/0 (socket says 172.21.15.38:33220) 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.196+0000 7fd670b26640 1 -- 172.21.15.38:0/2872202835 learned_addr learned my addr 172.21.15.38:0/2872202835 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd663fff640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd66c101890 0x7fd66c109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd671327640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 0x7fd66c109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd670b26640 1 -- 172.21.15.38:0/2872202835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd66c101890 msgr2=0x7fd66c109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd670b26640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd66c101890 0x7fd66c109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd670b26640 1 -- 172.21.15.38:0/2872202835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 msgr2=0x7fd66c109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd670b26640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 0x7fd66c109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd670b26640 1 -- 172.21.15.38:0/2872202835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66c10a430 con 0x7fd66c1006a0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd671327640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 0x7fd66c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd670b26640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c10b020 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fd668005ab0 tx=0x7fd668005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd661ffb640 1 -- 172.21.15.38:0/2872202835 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668041070 con 0x7fd66c1006a0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd661ffb640 1 -- 172.21.15.38:0/2872202835 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6680054e0 con 0x7fd66c1006a0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66c1a1b90 con 0x7fd66c1006a0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd661ffb640 1 -- 172.21.15.38:0/2872202835 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668034440 con 0x7fd66c1006a0 2024-04-22T01:15:22.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.197+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66c1a20e0 con 0x7fd66c1006a0 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.200+0000 7fd661ffb640 1 -- 172.21.15.38:0/2872202835 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd668044070 con 0x7fd66c1006a0 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.200+0000 7fd675973740 1 Processor -- start 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 -- start start 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d033dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d034390 0x55802d038e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d03b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55802cf98170 con 0x55802d039500 2024-04-22T01:15:22.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55802cf407c0 con 0x55802d0339a0 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd675973740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55802cf94130 con 0x55802d034390 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d03b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d03b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33226/0 (socket says 172.21.15.38:33226) 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 -- 172.21.15.38:0/2387485779 learned_addr learned my addr 172.21.15.38:0/2387485779 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd671327640 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d033dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd661ffb640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6540a2b90 0x7fd6540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd670b26640 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d034390 0x55802d038e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 -- 172.21.15.38:0/2387485779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d034390 msgr2=0x55802d038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d034390 0x55802d038e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 -- 172.21.15.38:0/2387485779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 msgr2=0x55802d033dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d033dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd663fff640 1 -- 172.21.15.38:0/2387485779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55802cf51e60 con 0x55802d039500 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd671327640 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d033dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd671327640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6540a2b90 0x7fd6540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.201+0000 7fd661ffb640 1 -- 172.21.15.38:0/2872202835 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd6680f1170 con 0x7fd66c1006a0 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd663fff640 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d03b990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00bce0 tx=0x7fd65c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120bea13d03a39f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd652ffd640 1 -- 172.21.15.38:0/2387485779 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c015070 con 0x55802d039500 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd652ffd640 1 -- 172.21.15.38:0/2387485779 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c01a940 con 0x55802d039500 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd652ffd640 1 -- 172.21.15.38:0/2387485779 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c011a30 con 0x55802d039500 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd675973740 1 -- 172.21.15.38:0/2387485779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 msgr2=0x55802d03b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd675973740 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d03b990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00bce0 tx=0x7fd65c00f730 comp rx=0 tx=0).stop 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.202+0000 7fd671327640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6540a2b90 0x7fd6540a5050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55802d034bf0 tx=0x7fd664004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 -- 172.21.15.38:0/2387485779 shutdown_connections 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d03b990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d034390 0x55802d038e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 --2- 172.21.15.38:0/2387485779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d033dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 -- 172.21.15.38:0/2387485779 >> 172.21.15.38:0/2387485779 conn(0x55802cffb4f0 msgr2=0x55802d02f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 -- 172.21.15.38:0/2387485779 shutdown_connections 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 -- 172.21.15.38:0/2387485779 wait complete. 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 -- start start 2024-04-22T01:15:22.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.203+0000 7fd675973740 1 Processor -- start 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d03df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d03e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d045430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55802d03c4b0 con 0x55802d039500 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55802cf53d40 con 0x55802d034390 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55802d03c630 con 0x55802d0339a0 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd675973740 10 client.0 _ll_register_callbacks cb 0x55802cf8f820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d045430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d045430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33232/0 (socket says 172.21.15.38:33232) 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 -- 172.21.15.38:0/4269450134 learned_addr learned my addr 172.21.15.38:0/4269450134 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd670b26640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d03df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd663fff640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d03e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 -- 172.21.15.38:0/4269450134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 msgr2=0x55802d03df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d03df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 -- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 msgr2=0x55802d03e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d03e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 -- 172.21.15.38:0/4269450134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802cd747e0 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd671327640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d045430 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd664009bb0 tx=0x7fd664004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162fec656fd29e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd64bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.204+0000 7fd64bfff640 1 -- 172.21.15.38:0/4269450134 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664023070 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd64bfff640 1 -- 172.21.15.38:0/4269450134 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd664012de0 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd65c01a940 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd64bfff640 1 -- 172.21.15.38:0/4269450134 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd66402f050 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 10 client.16422 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55802cdd4f70 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd64bfff640 1 -- 172.21.15.38:0/4269450134 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c01a940 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd64bfff640 1 -- 172.21.15.38:0/4269450134 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd664010590 con 0x55802d039500 2024-04-22T01:15:22.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 10 client.16422 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 10 client.16422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 20 client.16422 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0499f0 0x55802d04be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 4 client.16422 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55802cf6c450 con 0x55802d0499f0 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.205+0000 7fd670b26640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0499f0 0x55802d04be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.206+0000 7fd64bfff640 10 client.16422 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.206+0000 7fd670b26640 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0499f0 0x55802d04be50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af56a0e083602370 server_cookie=9cca22dc4247559c in_seq=0 out_seq=0 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.206+0000 7fd64bfff640 1 -- 172.21.15.38:0/4269450134 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55802cf6c450 con 0x55802d0499f0 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.206+0000 7fd64bfff640 10 client.16422 handle_command_reply: tid=0 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 client.16422 shutdown 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 msgr2=0x55802d045430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d045430 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd664009bb0 tx=0x7fd664004ab0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 shutdown_connections 2024-04-22T01:15:22.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0499f0 0x55802d04be50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d045430 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d03e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd64bfff640 0 client.16422 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 --2- 172.21.15.38:0/4269450134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d03df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 >> 172.21.15.38:0/4269450134 conn(0x55802cffb4f0 msgr2=0x55802d02e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 shutdown_connections 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 1 -- 172.21.15.38:0/4269450134 wait complete. 2024-04-22T01:15:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.207+0000 7fd675973740 20 client.16422 trim_cache size 0 max 16384 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 Processor -- start 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 -- start start 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d050750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 0x55802d0453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d039500 0x55802d050c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ac720 con 0x55802d034390 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ae260 con 0x55802d0339a0 2024-04-22T01:15:22.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd675973740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1acdf0 con 0x55802d039500 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.225+0000 7fd663fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 0x55802d0453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 0x55802d0453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33234/0 (socket says 172.21.15.38:33234) 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 -- 172.21.15.38:0/4278486687 learned_addr learned my addr 172.21.15.38:0/4278486687 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd670b26640 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d050750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 -- 172.21.15.38:0/4278486687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d039500 msgr2=0x55802d050c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d039500 0x55802d050c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 -- 172.21.15.38:0/4278486687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 msgr2=0x55802d050750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d050750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 -- 172.21.15.38:0/4278486687 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55802cc6c170 con 0x55802d034390 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd670b26640 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d050750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd663fff640 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 0x55802d0453f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00bc30 tx=0x7fd65c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3de7510d46c1ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd6527fc640 1 -- 172.21.15.38:0/4278486687 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c015050 con 0x55802d034390 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd6527fc640 1 -- 172.21.15.38:0/4278486687 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c028070 con 0x55802d034390 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.226+0000 7fd6527fc640 1 -- 172.21.15.38:0/4278486687 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c004d00 con 0x55802d034390 2024-04-22T01:15:22.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 -- 172.21.15.38:0/4278486687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 msgr2=0x55802d0453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 0x55802d0453f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00bc30 tx=0x7fd65c005fd0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 -- 172.21.15.38:0/4278486687 shutdown_connections 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d039500 0x55802d050c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d034390 0x55802d0453f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 --2- 172.21.15.38:0/4278486687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d0339a0 0x55802d050750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 -- 172.21.15.38:0/4278486687 >> 172.21.15.38:0/4278486687 conn(0x55802cffb4f0 msgr2=0x55802d02e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 -- 172.21.15.38:0/4278486687 shutdown_connections 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 -- 172.21.15.38:0/4278486687 wait complete. 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 -- start start 2024-04-22T01:15:22.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.227+0000 7fd675973740 1 Processor -- start 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d0450a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d0436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d043bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1adb90 con 0x55802d039500 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1ad4c0 con 0x55802d034390 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c1aa6e0 con 0x55802d0339a0 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd675973740 10 client.0 _ll_register_callbacks cb 0x7fd664009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d043bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d043bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33248/0 (socket says 172.21.15.38:33248) 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 -- 172.21.15.38:0/796672020 learned_addr learned my addr 172.21.15.38:0/796672020 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd663fff640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d0436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 -- 172.21.15.38:0/796672020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 msgr2=0x55802d0450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d0450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 -- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 msgr2=0x55802d0436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d0436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 -- 172.21.15.38:0/796672020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802d0441b0 con 0x55802d039500 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd663fff640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d0436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd671327640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d043bf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd664006010 tx=0x7fd6640114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2a097d3103bf69 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.228+0000 7fd64bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd64bfff640 1 -- 172.21.15.38:0/796672020 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664023070 con 0x55802d039500 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd64bfff640 1 -- 172.21.15.38:0/796672020 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd6640103d0 con 0x55802d039500 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd65c028070 con 0x55802d039500 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd64bfff640 1 -- 172.21.15.38:0/796672020 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd664040050 con 0x55802d039500 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 10 client.16428 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55802d0b0d30 con 0x55802d039500 2024-04-22T01:15:22.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd64bfff640 1 -- 172.21.15.38:0/796672020 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c028070 con 0x55802d039500 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd64bfff640 1 -- 172.21.15.38:0/796672020 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd66403b030 con 0x55802d039500 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 10 client.16428 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 20 client.16428 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0b16d0 0x55802d0b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 4 client.16428 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55802cfa56b0 con 0x55802d0b16d0 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.229+0000 7fd670b26640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0b16d0 0x55802d0b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.230+0000 7fd64bfff640 10 client.16428 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.230+0000 7fd670b26640 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0b16d0 0x55802d0b3af0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c5bfafa2405cf30 server_cookie=9b5ea6bb58b08f9d in_seq=0 out_seq=0 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd64bfff640 1 -- 172.21.15.38:0/796672020 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55802cfa56b0 con 0x55802d0b16d0 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd64bfff640 10 client.16428 handle_command_reply: tid=0 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 client.16428 shutdown 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 msgr2=0x55802d043bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d043bf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd664006010 tx=0x7fd6640114b0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 shutdown_connections 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55802d0b16d0 0x55802d0b3af0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55802d039500 0x55802d043bf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55802d034390 0x55802d0436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 --2- 172.21.15.38:0/796672020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55802d0339a0 0x55802d0450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 >> 172.21.15.38:0/796672020 conn(0x55802cffb4f0 msgr2=0x55802d032550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd64bfff640 0 client.16428 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 shutdown_connections 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 1 -- 172.21.15.38:0/796672020 wait complete. 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.231+0000 7fd675973740 20 client.16428 trim_cache size 0 max 16384 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:15:22.236 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T01:15:22.237 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "24739a33-d9cf-4663-abdc-dbe1acbba2ea", 2024-04-22T01:15:22.237 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T01:15:22.237 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:15:22.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6540a2b90 msgr2=0x7fd6540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6540a2b90 0x7fd6540a5050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55802d034bf0 tx=0x7fd664004a40 comp rx=0 tx=0).stop 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 msgr2=0x7fd66c10b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c10b020 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fd668005ab0 tx=0x7fd668005dc0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 shutdown_connections 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd6540a2b90 0x7fd6540a5050 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102250 0x7fd66c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd66c101890 0x7fd66c109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 --2- 172.21.15.38:0/2872202835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd66c1006a0 0x7fd66c10b020 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 >> 172.21.15.38:0/2872202835 conn(0x7fd66c0fbc30 msgr2=0x7fd66c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 shutdown_connections 2024-04-22T01:15:22.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.232+0000 7fd672d8a640 1 -- 172.21.15.38:0/2872202835 wait complete. 2024-04-22T01:15:22.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:15:22.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:15:22.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 -- 172.21.15.38:0/2049533928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d81006a0 msgr2=0x7f31d8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/2049533928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d81006a0 0x7f31d8100ac0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009900 tx=0x7f31d402bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 -- 172.21.15.38:0/2049533928 shutdown_connections 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/2049533928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d8102250 0x7f31d8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/2049533928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d8101890 0x7f31d8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/2049533928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d81006a0 0x7f31d8100ac0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 -- 172.21.15.38:0/2049533928 >> 172.21.15.38:0/2049533928 conn(0x7f31d80fbc30 msgr2=0x7f31d80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 -- 172.21.15.38:0/2049533928 shutdown_connections 2024-04-22T01:15:22.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 -- 172.21.15.38:0/2049533928 wait complete. 2024-04-22T01:15:22.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 Processor -- start 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.389+0000 7f31dfe6d640 1 -- start start 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81006a0 0x7f31d806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 0x7f31d819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d8102250 0x7f31d819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31d810c320 con 0x7f31d8101890 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31d810c1a0 con 0x7f31d81006a0 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31d810c4a0 con 0x7f31d8102250 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 0x7f31d819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 0x7f31d819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33260/0 (socket says 172.21.15.38:33260) 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 -- 172.21.15.38:0/3660243053 learned_addr learned my addr 172.21.15.38:0/3660243053 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31ddc09640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81006a0 0x7f31d806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31de40a640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d8102250 0x7f31d819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 -- 172.21.15.38:0/3660243053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d8102250 msgr2=0x7f31d819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d8102250 0x7f31d819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 -- 172.21.15.38:0/3660243053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81006a0 msgr2=0x7f31d806a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81006a0 0x7f31d806a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 -- 172.21.15.38:0/3660243053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d819f7b0 con 0x7f31d8101890 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dd408640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 0x7f31d819c880 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f31c800d770 tx=0x7f31c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31ceffd640 1 -- 172.21.15.38:0/3660243053 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8017070 con 0x7f31d8101890 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31ceffd640 1 -- 172.21.15.38:0/3660243053 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31c8005500 con 0x7f31d8101890 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31ceffd640 1 -- 172.21.15.38:0/3660243053 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8004e00 con 0x7f31d8101890 2024-04-22T01:15:22.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.390+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d819fa40 con 0x7f31d8101890 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.391+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d819ff90 con 0x7f31d8101890 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.393+0000 7f31ceffd640 1 -- 172.21.15.38:0/3660243053 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f31c8004fa0 con 0x7f31d8101890 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.393+0000 7f31e2a50740 1 Processor -- start 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.393+0000 7f31e2a50740 1 -- start start 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bbf50 0x55e2575bc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575c1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e2575c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31e2a50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e257521560 con 0x55e2575bc940 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31e2a50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e2574c9730 con 0x55e2575c1ab0 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31e2a50740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e25751d520 con 0x55e2575bbf50 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575c1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33276/0 (socket says 172.21.15.38:33276) 2024-04-22T01:15:22.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 -- 172.21.15.38:0/417059299 learned_addr learned my addr 172.21.15.38:0/417059299 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ceffd640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f31c40a2c10 0x7f31c40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31de40a640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f31c40a2c10 0x7f31c40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31dd408640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e2575c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31de40a640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bbf50 0x55e2575bc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ceffd640 1 -- 172.21.15.38:0/3660243053 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f31c80cee20 con 0x7f31d8101890 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 -- 172.21.15.38:0/417059299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bbf50 msgr2=0x55e2575bc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bbf50 0x55e2575bc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 -- 172.21.15.38:0/417059299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 msgr2=0x55e2575c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e2575c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 -- 172.21.15.38:0/417059299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2572ff7d0 con 0x55e2575bc940 2024-04-22T01:15:22.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31de40a640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bbf50 0x55e2575bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31dd408640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e2575c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.394+0000 7f31ddc09640 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575c1440 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009900 tx=0x7f31d4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1fc96c05432dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31de40a640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f31c40a2c10 0x7f31c40a50d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f31d0005fd0 tx=0x7f31d000a2b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31c3fff640 1 -- 172.21.15.38:0/417059299 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d402cc80 con 0x55e2575bc940 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31c3fff640 1 -- 172.21.15.38:0/417059299 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31d4005040 con 0x55e2575bc940 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31c3fff640 1 -- 172.21.15.38:0/417059299 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4004ab0 con 0x55e2575bc940 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 -- 172.21.15.38:0/417059299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 msgr2=0x55e2575c1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575c1440 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009900 tx=0x7f31d4005dd0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 -- 172.21.15.38:0/417059299 shutdown_connections 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e2575c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575c1440 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 --2- 172.21.15.38:0/417059299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bbf50 0x55e2575bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.395+0000 7f31e2a50740 1 -- 172.21.15.38:0/417059299 >> 172.21.15.38:0/417059299 conn(0x55e2575b74c0 msgr2=0x55e2575b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 -- 172.21.15.38:0/417059299 shutdown_connections 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 -- 172.21.15.38:0/417059299 wait complete. 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 -- start start 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 Processor -- start 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575c6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e2575c6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 0x55e2575cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e2575c4a00 con 0x55e2575c1ab0 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e2574dce00 con 0x55e2575bbf50 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e2575c4b80 con 0x55e2575bc940 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31e2a50740 10 client.0 _ll_register_callbacks cb 0x55e257517eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31de40a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 0x55e2575cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31de40a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 0x55e2575cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33282/0 (socket says 172.21.15.38:33282) 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.396+0000 7f31de40a640 1 -- 172.21.15.38:0/943339061 learned_addr learned my addr 172.21.15.38:0/943339061 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31de40a640 1 -- 172.21.15.38:0/943339061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 msgr2=0x55e2575c6970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31de40a640 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e2575c6970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31de40a640 1 -- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 msgr2=0x55e2575c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31ddc09640 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31de40a640 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575c6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31de40a640 1 -- 172.21.15.38:0/943339061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2572353f0 con 0x55e2575c1ab0 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31de40a640 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 0x55e2575cd9b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f31d00119e0 tx=0x7f31d0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1761a9fdbc38770f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31c17fa640 1 -- 172.21.15.38:0/943339061 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d0021070 con 0x55e2575c1ab0 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31c17fa640 1 -- 172.21.15.38:0/943339061 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f31d0010d60 con 0x55e2575c1ab0 2024-04-22T01:15:22.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31d4005040 con 0x55e2575c1ab0 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31c17fa640 1 -- 172.21.15.38:0/943339061 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31d002d050 con 0x55e2575c1ab0 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31e2a50740 10 client.16440 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2571f50f0 con 0x55e2575c1ab0 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.397+0000 7f31c17fa640 1 -- 172.21.15.38:0/943339061 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31d4005040 con 0x55e2575c1ab0 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31c17fa640 1 -- 172.21.15.38:0/943339061 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f31d001a040 con 0x55e2575c1ab0 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31e2a50740 10 client.16440 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31e2a50740 10 client.16440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31e2a50740 20 client.16440 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31e2a50740 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e2575d2280 0x55e2575d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31e2a50740 4 client.16440 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e25752eaa0 con 0x55e2575d2280 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31ddc09640 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e2575d2280 0x55e2575d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31c17fa640 10 client.16440 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.398+0000 7f31ddc09640 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e2575d2280 0x55e2575d4740 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa803ac20726152a server_cookie=93329c1372817aab in_seq=0 out_seq=0 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31c17fa640 1 -- 172.21.15.38:0/943339061 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e25752eaa0 con 0x55e2575d2280 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31c17fa640 10 client.16440 handle_command_reply: tid=0 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 client.16440 shutdown 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 msgr2=0x55e2575cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 0x55e2575cd9b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f31d00119e0 tx=0x7f31d0004ab0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 shutdown_connections 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e2575d2280 0x55e2575d4740 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575c1ab0 0x55e2575cd9b0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e2575c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 --2- 172.21.15.38:0/943339061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575c6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31c17fa640 0 client.16440 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 >> 172.21.15.38:0/943339061 conn(0x55e2575b74c0 msgr2=0x55e2575b66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 shutdown_connections 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 1 -- 172.21.15.38:0/943339061 wait complete. 2024-04-22T01:15:22.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.399+0000 7f31e2a50740 20 client.16440 trim_cache size 0 max 16384 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 Processor -- start 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 -- start start 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575c1ab0 0x55e2575b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81aa440 con 0x55e2575bc940 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81abf80 con 0x55e2575bbf50 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31e2a50740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81aab10 con 0x55e2575c1ab0 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31dd408640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31dd408640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575b5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33296/0 (socket says 172.21.15.38:33296) 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.417+0000 7f31dd408640 1 -- 172.21.15.38:0/2457338512 learned_addr learned my addr 172.21.15.38:0/2457338512 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31ddc09640 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31de40a640 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575c1ab0 0x55e2575b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31dd408640 1 -- 172.21.15.38:0/2457338512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575c1ab0 msgr2=0x55e2575b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31dd408640 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575c1ab0 0x55e2575b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31dd408640 1 -- 172.21.15.38:0/2457338512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 msgr2=0x55e2575b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31dd408640 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575b2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31dd408640 1 -- 172.21.15.38:0/2457338512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2575d0bd0 con 0x55e2575bc940 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31de40a640 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575c1ab0 0x55e2575b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31dd408640 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575b5e50 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f31c80976d0 tx=0x7f31c8016310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ca81dc9b0117f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31c37fe640 1 -- 172.21.15.38:0/2457338512 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c809b970 con 0x55e2575bc940 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31c37fe640 1 -- 172.21.15.38:0/2457338512 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31c809b270 con 0x55e2575bc940 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.418+0000 7f31c37fe640 1 -- 172.21.15.38:0/2457338512 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c80978e0 con 0x55e2575bc940 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- 172.21.15.38:0/2457338512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 msgr2=0x55e2575b5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575b5e50 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f31c80976d0 tx=0x7f31c8016310 comp rx=0 tx=0).stop 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- 172.21.15.38:0/2457338512 shutdown_connections 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575c1ab0 0x55e2575b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bc940 0x55e2575b5e50 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- 172.21.15.38:0/2457338512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575bbf50 0x55e2575b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- 172.21.15.38:0/2457338512 >> 172.21.15.38:0/2457338512 conn(0x55e2575b74c0 msgr2=0x55e2575bde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- 172.21.15.38:0/2457338512 shutdown_connections 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- 172.21.15.38:0/2457338512 wait complete. 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- start start 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 Processor -- start 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 0x55e257633990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e257633ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e25763af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81ab8b0 con 0x55e2575bbf50 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81ab1e0 con 0x55e2575c1ab0 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81a8400 con 0x55e2575bc940 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31e2a50740 10 client.0 _ll_register_callbacks cb 0x7f31d000fea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31ddc09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 0x55e257633990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31ddc09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 0x55e257633990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33306/0 (socket says 172.21.15.38:33306) 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.419+0000 7f31ddc09640 1 -- 172.21.15.38:0/1649146522 learned_addr learned my addr 172.21.15.38:0/1649146522 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31dd408640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e257633ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31de40a640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e25763af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31ddc09640 1 -- 172.21.15.38:0/1649146522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 msgr2=0x55e257633ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31ddc09640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e257633ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31ddc09640 1 -- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 msgr2=0x55e25763af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31ddc09640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e25763af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31ddc09640 1 -- 172.21.15.38:0/1649146522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e25763b500 con 0x55e2575bbf50 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31de40a640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e25763af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31ddc09640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 0x55e257633990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f31d402cfd0 tx=0x7f31d4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc245dff7ec31c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31c17fa640 1 -- 172.21.15.38:0/1649146522 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d404a070 con 0x55e2575bbf50 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31c17fa640 1 -- 172.21.15.38:0/1649146522 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f31d4005450 con 0x55e2575bbf50 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31c809b270 con 0x55e2575bbf50 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31c17fa640 1 -- 172.21.15.38:0/1649146522 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31d4005650 con 0x55e2575bbf50 2024-04-22T01:15:22.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31e2a50740 10 client.16446 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e25763c7f0 con 0x55e2575bbf50 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.420+0000 7f31c17fa640 1 -- 172.21.15.38:0/1649146522 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31c809b270 con 0x55e2575bbf50 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31c17fa640 1 -- 172.21.15.38:0/1649146522 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f31d404e3a0 con 0x55e2575bbf50 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 10 client.16446 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 10 client.16446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 20 client.16446 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e25763cdb0 0x55e25763f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 4 client.16446 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e257528df0 con 0x55e25763cdb0 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31dd408640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e25763cdb0 0x55e25763f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31c17fa640 10 client.16446 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31dd408640 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e25763cdb0 0x55e25763f1a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55442afa87b5c59 server_cookie=2a6ff1b08d09dd59 in_seq=0 out_seq=0 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31c17fa640 1 -- 172.21.15.38:0/1649146522 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e257528df0 con 0x55e25763cdb0 2024-04-22T01:15:22.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31c17fa640 10 client.16446 handle_command_reply: tid=0 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 1 client.16446 shutdown 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 msgr2=0x55e257633990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.421+0000 7f31e2a50740 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 0x55e257633990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f31d402cfd0 tx=0x7f31d4005d50 comp rx=0 tx=0).stop 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 shutdown_connections 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55e25763cdb0 0x55e25763f1a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2575c1ab0 0x55e25763af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31c17fa640 0 client.16446 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e2575bc940 0x55e257633ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 --2- 172.21.15.38:0/1649146522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e2575bbf50 0x55e257633990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 >> 172.21.15.38:0/1649146522 conn(0x55e2575b74c0 msgr2=0x55e2575ba720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 shutdown_connections 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 1 -- 172.21.15.38:0/1649146522 wait complete. 2024-04-22T01:15:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31e2a50740 20 client.16446 trim_cache size 0 max 16384 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "24739a33-d9cf-4663-abdc-dbe1acbba2ea": { 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "24739a33-d9cf-4663-abdc-dbe1acbba2ea", 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:22.923 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f31c40a2c10 msgr2=0x7f31c40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f31c40a2c10 0x7f31c40a50d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f31d0005fd0 tx=0x7f31d000a2b0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 msgr2=0x7f31d819c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 0x7f31d819c880 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f31c800d770 tx=0x7f31c800dc30 comp rx=0 tx=0).stop 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 shutdown_connections 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f31c40a2c10 0x7f31c40a50d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31d8102250 0x7f31d819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31d8101890 0x7f31d819c880 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 --2- 172.21.15.38:0/3660243053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31d81006a0 0x7f31d806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 >> 172.21.15.38:0/3660243053 conn(0x7f31d80fbc30 msgr2=0x7f31d8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 shutdown_connections 2024-04-22T01:15:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:22.422+0000 7f31dfe6d640 1 -- 172.21.15.38:0/3660243053 wait complete. 2024-04-22T01:15:22.925 INFO:tasks.cephfs.filesystem:scrub status for tag:24739a33-d9cf-4663-abdc-dbe1acbba2ea - {'path': '/', 'tag': '24739a33-d9cf-4663-abdc-dbe1acbba2ea', 'options': 'recursive,force'} 2024-04-22T01:15:23.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:22 smithi203 ceph-mon[31825]: scrub queued for path: / 2024-04-22T01:15:23.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:22 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:15:23.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:22 smithi203 ceph-mon[31825]: scrub summary: active paths [/] 2024-04-22T01:15:23.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:22 smithi203 ceph-mon[31825]: scrub summary: idle+waiting paths [/] 2024-04-22T01:15:23.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:22 smithi111 ceph-mon[31057]: scrub queued for path: / 2024-04-22T01:15:23.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:22 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:15:23.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:22 smithi111 ceph-mon[31057]: scrub summary: active paths [/] 2024-04-22T01:15:23.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:22 smithi111 ceph-mon[31057]: scrub summary: idle+waiting paths [/] 2024-04-22T01:15:23.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:22 smithi038 ceph-mon[28043]: scrub queued for path: / 2024-04-22T01:15:23.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:22 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:15:23.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:22 smithi038 ceph-mon[28043]: scrub summary: active paths [/] 2024-04-22T01:15:23.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:22 smithi038 ceph-mon[28043]: scrub summary: idle+waiting paths [/] 2024-04-22T01:15:24.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:23 smithi203 ceph-mon[31825]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 746 MiB/s wr, 323 op/s 2024-04-22T01:15:24.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:23 smithi111 ceph-mon[31057]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 746 MiB/s wr, 323 op/s 2024-04-22T01:15:24.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:23 smithi038 ceph-mon[28043]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 746 MiB/s wr, 323 op/s 2024-04-22T01:15:26.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:25 smithi203 ceph-mon[31825]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 787 MiB/s wr, 335 op/s 2024-04-22T01:15:26.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:25 smithi111 ceph-mon[31057]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 787 MiB/s wr, 335 op/s 2024-04-22T01:15:26.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:25 smithi038 ceph-mon[28043]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 787 MiB/s wr, 335 op/s 2024-04-22T01:15:26.712 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:26.712 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 readers = 8952630.00 kB/sec 2024-04-22T01:15:26.712 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 readers = 8950833.59 kB/sec 2024-04-22T01:15:26.712 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 8952630.00 kB/sec 2024-04-22T01:15:26.713 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 8952630.00 kB/sec 2024-04-22T01:15:26.713 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 8952630.00 kB/sec 2024-04-22T01:15:26.713 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 10485760.00 kB 2024-04-22T01:15:27.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:26 smithi203 ceph-mon[31825]: scrub summary: idle 2024-04-22T01:15:27.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:26 smithi111 ceph-mon[31057]: scrub summary: idle 2024-04-22T01:15:27.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:26 smithi038 ceph-mon[28043]: scrub summary: idle 2024-04-22T01:15:28.025 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:27 smithi203 ceph-mon[31825]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 773 MiB/s wr, 331 op/s 2024-04-22T01:15:28.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:27 smithi111 ceph-mon[31057]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 773 MiB/s wr, 331 op/s 2024-04-22T01:15:28.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:27 smithi038 ceph-mon[28043]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 773 MiB/s wr, 331 op/s 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: Children see throughput for 1 re-readers = 8971819.00 kB/sec 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: Parent sees throughput for 1 re-readers = 8969939.77 kB/sec 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: Min throughput per process = 8971819.00 kB/sec 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: Max throughput per process = 8971819.00 kB/sec 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: Avg throughput per process = 8971819.00 kB/sec 2024-04-22T01:15:29.967 INFO:tasks.workunit.client.0.smithi038.stdout: Min xfer = 10485760.00 kB 2024-04-22T01:15:30.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:29 smithi203 ceph-mon[31825]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 646 MiB/s wr, 204 op/s 2024-04-22T01:15:30.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:29 smithi111 ceph-mon[31057]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 646 MiB/s wr, 204 op/s 2024-04-22T01:15:30.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:29 smithi038 ceph-mon[28043]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 646 MiB/s wr, 204 op/s 2024-04-22T01:15:32.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:31 smithi203 ceph-mon[31825]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 439 MiB/s wr, 151 op/s 2024-04-22T01:15:32.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:31 smithi111 ceph-mon[31057]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 439 MiB/s wr, 151 op/s 2024-04-22T01:15:32.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:31 smithi038 ceph-mon[28043]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 439 MiB/s wr, 151 op/s 2024-04-22T01:15:34.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:33 smithi203 ceph-mon[31825]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 246 MiB/s wr, 75 op/s 2024-04-22T01:15:34.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:33 smithi111 ceph-mon[31057]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 246 MiB/s wr, 75 op/s 2024-04-22T01:15:34.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:33 smithi038 ceph-mon[28043]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 246 MiB/s wr, 75 op/s 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3933151012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0107480 msgr2=0x7fa4d01098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3933151012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0107480 0x7fa4d01098d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009ca0 tx=0x7fa4c802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3933151012 shutdown_connections 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3933151012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0107480 0x7fa4d01098d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3933151012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4d0104b50 0x7fa4d0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3933151012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0102220 0x7fa4d0104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3933151012 >> 172.21.15.38:0/3933151012 conn(0x7fa4d00fbc10 msgr2=0x7fa4d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3933151012 shutdown_connections 2024-04-22T01:15:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3933151012 wait complete. 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.736+0000 7fa4d50e2640 1 Processor -- start 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 -- start start 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4d0102220 0x7fa4d0101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 0x7fa4d01001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0107480 0x7fa4d0100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d010d420 con 0x7fa4d0104b50 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d010d2a0 con 0x7fa4d0107480 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4d50e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d010d5a0 con 0x7fa4d0102220 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 0x7fa4d01001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 0x7fa4d01001d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40522/0 (socket says 172.21.15.38:40522) 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3565779048 learned_addr learned my addr 172.21.15.38:0/3565779048 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0107480 0x7fa4d0100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4d0102220 0x7fa4d0101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3565779048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4d0102220 msgr2=0x7fa4d0101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4d0102220 0x7fa4d0101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3565779048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0107480 msgr2=0x7fa4d0100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0107480 0x7fa4d0100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3565779048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d0100fa0 con 0x7fa4d0104b50 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0107480 0x7fa4d0100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.737+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 0x7fa4d01001d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d770 tx=0x7fa4c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.738+0000 7fa4b7fff640 1 -- 172.21.15.38:0/3565779048 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4017070 con 0x7fa4d0104b50 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.738+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d01a1b00 con 0x7fa4d0104b50 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.738+0000 7fa4b7fff640 1 -- 172.21.15.38:0/3565779048 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c4009c30 con 0x7fa4d0104b50 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.738+0000 7fa4b7fff640 1 -- 172.21.15.38:0/3565779048 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c400f620 con 0x7fa4d0104b50 2024-04-22T01:15:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.738+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d01a20a0 con 0x7fa4d0104b50 2024-04-22T01:15:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4b7fff640 1 -- 172.21.15.38:0/3565779048 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa4c401a070 con 0x7fa4d0104b50 2024-04-22T01:15:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 Processor -- start 2024-04-22T01:15:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 -- start start 2024-04-22T01:15:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a74150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a79220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 0x557fb4a7bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557fb49d8b70 con 0x557fb4a79890 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557fb4981470 con 0x557fb4a74720 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.740+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557fb49d4b30 con 0x557fb4a73d30 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 0x557fb4a7bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 0x557fb4a7bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40536/0 (socket says 172.21.15.38:40536) 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3732628835 learned_addr learned my addr 172.21.15.38:0/3732628835 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a74150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a79220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4b7fff640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4b80a2c30 0x7fa4b80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3732628835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 msgr2=0x557fb4a74150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a74150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3732628835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 msgr2=0x557fb4a79220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a79220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 -- 172.21.15.38:0/3732628835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fb46ed210 con 0x557fb4a79890 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4b80a2c30 0x7fa4b80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4b7fff640 1 -- 172.21.15.38:0/3565779048 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa4c40cf030 con 0x7fa4d0104b50 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 0x557fb4a7bd20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40b1a70 tx=0x7fa4c40af620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4b8861aaa106fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4b57fa640 1 -- 172.21.15.38:0/3732628835 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4021030 con 0x557fb4a79890 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4b57fa640 1 -- 172.21.15.38:0/3732628835 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c409bbc0 con 0x557fb4a79890 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.741+0000 7fa4b57fa640 1 -- 172.21.15.38:0/3732628835 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c409ae20 con 0x557fb4a79890 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 -- 172.21.15.38:0/3732628835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 msgr2=0x557fb4a7bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4b80a2c30 0x7fa4b80a50f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa4c802c630 tx=0x7fa4c802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 0x557fb4a7bd20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40b1a70 tx=0x7fa4c40af620 comp rx=0 tx=0).stop 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 -- 172.21.15.38:0/3732628835 shutdown_connections 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a79890 0x557fb4a7bd20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a79220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/3732628835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a74150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 -- 172.21.15.38:0/3732628835 >> 172.21.15.38:0/3732628835 conn(0x557fb4a6f2a0 msgr2=0x557fb4a6f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 -- 172.21.15.38:0/3732628835 shutdown_connections 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 -- 172.21.15.38:0/3732628835 wait complete. 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.742+0000 7fa4d7da9740 1 -- start start 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 Processor -- start 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a7e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a7e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a79890 0x557fb4a855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557fb4a7c7e0 con 0x557fb4a73d30 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557fb4994a10 con 0x557fb4a74720 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557fb4a7c960 con 0x557fb4a79890 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4d7da9740 10 client.0 _ll_register_callbacks cb 0x557fb49cfb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a7e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a7e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40540/0 (socket says 172.21.15.38:40540) 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 -- 172.21.15.38:0/2561922167 learned_addr learned my addr 172.21.15.38:0/2561922167 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a7e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 -- 172.21.15.38:0/2561922167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a79890 msgr2=0x557fb4a855e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a79890 0x557fb4a855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 -- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 msgr2=0x557fb4a7e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a7e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 -- 172.21.15.38:0/2561922167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fb47b9620 con 0x557fb4a73d30 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a7e060 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0012440 tx=0x7fa4c0012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45612c75c7d8483a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.743+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/2561922167 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0005220 con 0x557fb4a73d30 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/2561922167 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa4c00053c0 con 0x557fb4a73d30 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c409bbc0 con 0x557fb4a73d30 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/2561922167 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa4c00055c0 con 0x557fb4a73d30 2024-04-22T01:15:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 10 client.16458 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557fb4a88a20 con 0x557fb4a73d30 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/2561922167 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c409bbc0 con 0x557fb4a73d30 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/2561922167 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa4c000f040 con 0x557fb4a73d30 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 10 client.16458 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 10 client.16458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 20 client.16458 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4a8a160 0x557fb4a8c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 4 client.16458 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fb49e60b0 con 0x557fb4a8a160 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.744+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4a8a160 0x557fb4a8c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.745+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4a8a160 0x557fb4a8c5d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9efdfadfeccd9361 server_cookie=923412b668a32dc4 in_seq=0 out_seq=0 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.745+0000 7fa4b2ff5640 10 client.16458 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.745+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/2561922167 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557fb49e60b0 con 0x557fb4a8a160 2024-04-22T01:15:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.745+0000 7fa4b2ff5640 10 client.16458 handle_command_reply: tid=0 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.745+0000 7fa4d7da9740 1 client.16458 shutdown 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 msgr2=0x557fb4a7e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a7e060 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0012440 tx=0x7fa4c0012900 comp rx=0 tx=0).stop 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 shutdown_connections 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4a8a160 0x557fb4a8c5d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a79890 0x557fb4a855e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4b2ff5640 0 client.16458 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a74720 0x557fb4a7e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/2561922167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a7e060 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 >> 172.21.15.38:0/2561922167 conn(0x557fb4a6f2a0 msgr2=0x557fb4a6e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 shutdown_connections 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 1 -- 172.21.15.38:0/2561922167 wait complete. 2024-04-22T01:15:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.746+0000 7fa4d7da9740 20 client.16458 trim_cache size 0 max 16384 2024-04-22T01:15:34.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 Processor -- start 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 -- start start 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a72c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 0x557fb4a855a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4a90ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01ac790 con 0x557fb4a74720 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01ae2d0 con 0x557fb4a79890 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01ace60 con 0x557fb4a73d30 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 0x557fb4a855a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 0x557fb4a855a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40544/0 (socket says 172.21.15.38:40544) 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 -- 172.21.15.38:0/1673504664 learned_addr learned my addr 172.21.15.38:0/1673504664 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4a90ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 -- 172.21.15.38:0/1673504664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 msgr2=0x557fb4a72c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a72c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 -- 172.21.15.38:0/1673504664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 msgr2=0x557fb4a90ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4a90ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 -- 172.21.15.38:0/1673504664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fb4a74b00 con 0x557fb4a74720 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 0x557fb4a855a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40a42f0 tx=0x7fa4c40d2ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e8e5e290f5adf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4b4ff9640 1 -- 172.21.15.38:0/1673504664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4021040 con 0x557fb4a74720 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4b4ff9640 1 -- 172.21.15.38:0/1673504664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c409bbc0 con 0x557fb4a74720 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.761+0000 7fa4b4ff9640 1 -- 172.21.15.38:0/1673504664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40a0a00 con 0x557fb4a74720 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1673504664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 msgr2=0x557fb4a855a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 0x557fb4a855a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40a42f0 tx=0x7fa4c40d2ab0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1673504664 shutdown_connections 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4a90ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a74720 0x557fb4a855a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1673504664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a73d30 0x557fb4a72c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1673504664 >> 172.21.15.38:0/1673504664 conn(0x557fb4a6f2a0 msgr2=0x557fb4a72500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1673504664 shutdown_connections 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1673504664 wait complete. 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 -- start start 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.762+0000 7fa4d7da9740 1 Processor -- start 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a825d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a74720 0x557fb4a82b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4aee740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01adc00 con 0x557fb4a73d30 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01ad530 con 0x557fb4a79890 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01aa750 con 0x557fb4a74720 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 10 client.0 _ll_register_callbacks cb 0x7fa4c0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a825d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a825d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40546/0 (socket says 172.21.15.38:40546) 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 -- 172.21.15.38:0/1079177098 learned_addr learned my addr 172.21.15.38:0/1079177098 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4aee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 -- 172.21.15.38:0/1079177098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a74720 msgr2=0x557fb4a82b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a74720 0x557fb4a82b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 -- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 msgr2=0x557fb4aee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4aee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 -- 172.21.15.38:0/1079177098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fb4aeed30 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4cf59e640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4aee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4ced9d640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a825d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00059d0 tx=0x7fa4c0013f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc679373f67d962 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/1079177098 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c002f060 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/1079177098 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa4c002a630 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4c409bbc0 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/1079177098 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa4c0036050 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 10 client.16464 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.763+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557fb4af00e0 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/1079177098 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c409bbc0 con 0x557fb4a73d30 2024-04-22T01:15:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/1079177098 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa4c0029070 con 0x557fb4a73d30 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4d7da9740 10 client.16464 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4d7da9740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4d7da9740 20 client.16464 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4af0b10 0x557fb4af2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4d7da9740 4 client.16464 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557fb49e0400 con 0x557fb4af0b10 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4af0b10 0x557fb4af2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4b2ff5640 10 client.16464 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.764+0000 7fa4ce59c640 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4af0b10 0x557fb4af2f30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51eb51a118cc562a server_cookie=3c27b195335996b8 in_seq=0 out_seq=0 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4b2ff5640 1 -- 172.21.15.38:0/1079177098 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557fb49e0400 con 0x557fb4af0b10 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4b2ff5640 10 client.16464 handle_command_reply: tid=0 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 client.16464 shutdown 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 msgr2=0x557fb4a825d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a825d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00059d0 tx=0x7fa4c0013f00 comp rx=0 tx=0).stop 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 shutdown_connections 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x557fb4af0b10 0x557fb4af2f30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557fb4a79890 0x557fb4aee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557fb4a74720 0x557fb4a82b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 --2- 172.21.15.38:0/1079177098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557fb4a73d30 0x557fb4a825d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4b2ff5640 0 client.16464 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 >> 172.21.15.38:0/1079177098 conn(0x557fb4a6f2a0 msgr2=0x557fb4a6dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 shutdown_connections 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.776+0000 7fa4d7da9740 1 -- 172.21.15.38:0/1079177098 wait complete. 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d7da9740 20 client.16464 trim_cache size 0 max 16384 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T01:15:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4b80a2c30 msgr2=0x7fa4b80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4b80a2c30 0x7fa4b80a50f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa4c802c630 tx=0x7fa4c802e040 comp rx=0 tx=0).stop 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 msgr2=0x7fa4d01001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 0x7fa4d01001d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d770 tx=0x7fa4c400dc30 comp rx=0 tx=0).stop 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 shutdown_connections 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa4b80a2c30 0x7fa4b80a50f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4d0107480 0x7fa4d0100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4d0104b50 0x7fa4d01001d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 --2- 172.21.15.38:0/3565779048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4d0102220 0x7fa4d0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 >> 172.21.15.38:0/3565779048 conn(0x7fa4d00fbc10 msgr2=0x7fa4d0102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 shutdown_connections 2024-04-22T01:15:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T01:15:34.777+0000 7fa4d50e2640 1 -- 172.21.15.38:0/3565779048 wait complete. 2024-04-22T01:15:36.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:35 smithi203 ceph-mon[31825]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 117 MiB/s wr, 43 op/s 2024-04-22T01:15:36.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:35 smithi111 ceph-mon[31057]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 117 MiB/s wr, 43 op/s 2024-04-22T01:15:36.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:35 smithi038 ceph-mon[28043]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 117 MiB/s wr, 43 op/s 2024-04-22T01:15:36.757 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:36.757 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:36.757 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T01:15:36.757 INFO:tasks.workunit.client.0.smithi038.stdout:iozone test complete. 2024-04-22T01:15:36.757 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T01:15:36.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T01:15:36.808 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-04-22T01:15:36.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T01:15:37.339 DEBUG:teuthology.parallel:result is None 2024-04-22T01:15:37.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T01:15:37.389 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T01:15:37.389 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-22T01:15:37.400 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:15:37.412 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-22T01:15:37.422 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-22T01:15:38.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:37 smithi203 ceph-mon[31825]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 32 MiB/s wr, 19 op/s 2024-04-22T01:15:38.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:37 smithi111 ceph-mon[31057]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 32 MiB/s wr, 19 op/s 2024-04-22T01:15:38.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:37 smithi038 ceph-mon[28043]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 32 MiB/s wr, 19 op/s 2024-04-22T01:15:40.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:39 smithi203 ceph-mon[31825]: pgmap v454: 129 pgs: 129 active+clean; 9.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 33 op/s 2024-04-22T01:15:40.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:39 smithi111 ceph-mon[31057]: pgmap v454: 129 pgs: 129 active+clean; 9.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 33 op/s 2024-04-22T01:15:40.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:39 smithi038 ceph-mon[28043]: pgmap v454: 129 pgs: 129 active+clean; 9.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 33 op/s 2024-04-22T01:15:42.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:41 smithi203 ceph-mon[31825]: pgmap v455: 129 pgs: 129 active+clean; 4.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.9 KiB/s wr, 131 op/s 2024-04-22T01:15:42.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:41 smithi111 ceph-mon[31057]: pgmap v455: 129 pgs: 129 active+clean; 4.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.9 KiB/s wr, 131 op/s 2024-04-22T01:15:42.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:41 smithi038 ceph-mon[28043]: pgmap v455: 129 pgs: 129 active+clean; 4.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.9 KiB/s wr, 131 op/s 2024-04-22T01:15:44.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:43 smithi203 ceph-mon[31825]: pgmap v456: 129 pgs: 129 active+clean; 1.0 GiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.9 KiB/s wr, 206 op/s 2024-04-22T01:15:44.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:43 smithi111 ceph-mon[31057]: pgmap v456: 129 pgs: 129 active+clean; 1.0 GiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.9 KiB/s wr, 206 op/s 2024-04-22T01:15:44.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:43 smithi038 ceph-mon[28043]: pgmap v456: 129 pgs: 129 active+clean; 1.0 GiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.9 KiB/s wr, 206 op/s 2024-04-22T01:15:46.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:45 smithi111 ceph-mon[31057]: pgmap v457: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 228 op/s 2024-04-22T01:15:46.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:45 smithi038 ceph-mon[28043]: pgmap v457: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 228 op/s 2024-04-22T01:15:46.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:45 smithi203 ceph-mon[31825]: pgmap v457: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 228 op/s 2024-04-22T01:15:48.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:47 smithi111 ceph-mon[31057]: pgmap v458: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 228 op/s 2024-04-22T01:15:48.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:47 smithi038 ceph-mon[28043]: pgmap v458: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 228 op/s 2024-04-22T01:15:48.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:47 smithi203 ceph-mon[31825]: pgmap v458: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 228 op/s 2024-04-22T01:15:50.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:49 smithi111 ceph-mon[31057]: pgmap v459: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.6 KiB/s wr, 229 op/s 2024-04-22T01:15:50.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:49 smithi038 ceph-mon[28043]: pgmap v459: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.6 KiB/s wr, 229 op/s 2024-04-22T01:15:50.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:49 smithi203 ceph-mon[31825]: pgmap v459: 129 pgs: 129 active+clean; 1022 KiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.6 KiB/s wr, 229 op/s 2024-04-22T01:15:52.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:51 smithi111 ceph-mon[31057]: pgmap v460: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.3 KiB/s wr, 208 op/s 2024-04-22T01:15:52.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:51 smithi038 ceph-mon[28043]: pgmap v460: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.3 KiB/s wr, 208 op/s 2024-04-22T01:15:52.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:51 smithi203 ceph-mon[31825]: pgmap v460: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.3 KiB/s wr, 208 op/s 2024-04-22T01:15:52.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T01:15:52.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T01:15:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 -- 172.21.15.38:0/132705286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff81074c0 msgr2=0x7efff81098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 --2- 172.21.15.38:0/132705286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff81074c0 0x7efff81098b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7efff0009900 tx=0x7efff002bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 -- 172.21.15.38:0/132705286 shutdown_connections 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 --2- 172.21.15.38:0/132705286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff81074c0 0x7efff81098b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 --2- 172.21.15.38:0/132705286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff8104b90 0x7efff8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 --2- 172.21.15.38:0/132705286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff8102260 0x7efff8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 -- 172.21.15.38:0/132705286 >> 172.21.15.38:0/132705286 conn(0x7efff80fbbf0 msgr2=0x7efff80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 -- 172.21.15.38:0/132705286 shutdown_connections 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 -- 172.21.15.38:0/132705286 wait complete. 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.045+0000 7efffd6c9640 1 Processor -- start 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 -- start start 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff8102260 0x7efff8101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 0x7efff8100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81074c0 0x7efff8100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efff810d470 con 0x7efff8104b90 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efff810d2f0 con 0x7efff8102260 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efffd6c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efff810d5f0 con 0x7efff81074c0 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 0x7efff8100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 0x7efff8100210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60910/0 (socket says 172.21.15.38:60910) 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 -- 172.21.15.38:0/205679075 learned_addr learned my addr 172.21.15.38:0/205679075 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff6ffd640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff8102260 0x7efff8101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff77fe640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81074c0 0x7efff8100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 -- 172.21.15.38:0/205679075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81074c0 msgr2=0x7efff8100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81074c0 0x7efff8100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 -- 172.21.15.38:0/205679075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff8102260 msgr2=0x7efff8101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff8102260 0x7efff8101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.046+0000 7efff67fc640 1 -- 172.21.15.38:0/205679075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff8100fe0 con 0x7efff8104b90 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.047+0000 7efff67fc640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 0x7efff8100210 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7effec00d740 tx=0x7effec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.047+0000 7effdffff640 1 -- 172.21.15.38:0/205679075 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec017070 con 0x7efff8104b90 2024-04-22T01:15:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.047+0000 7effdffff640 1 -- 172.21.15.38:0/205679075 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effec00de50 con 0x7efff8104b90 2024-04-22T01:15:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.047+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff81a1bd0 con 0x7efff8104b90 2024-04-22T01:15:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.047+0000 7effdffff640 1 -- 172.21.15.38:0/205679075 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec004e00 con 0x7efff8104b90 2024-04-22T01:15:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.047+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff81a2120 con 0x7efff8104b90 2024-04-22T01:15:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7effdffff640 1 -- 172.21.15.38:0/205679075 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7effec004fa0 con 0x7efff8104b90 2024-04-22T01:15:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 Processor -- start 2024-04-22T01:15:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 -- start start 2024-04-22T01:15:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 0x55d0cefc1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc1940 0x55d0cefc6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefc8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0cef26560 con 0x55d0cefc0f50 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d0ceece730 con 0x55d0cefc1940 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.049+0000 7f000038f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0cef22520 con 0x55d0cefc6ab0 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 0x55d0cefc1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 0x55d0cefc1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60912/0 (socket says 172.21.15.38:60912) 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 -- 172.21.15.38:0/1585303178 learned_addr learned my addr 172.21.15.38:0/1585303178 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff67fc640 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefc8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff6ffd640 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc1940 0x55d0cefc6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7effdffff640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00a2c10 0x7effe00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 -- 172.21.15.38:0/1585303178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 msgr2=0x55d0cefc8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefc8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 -- 172.21.15.38:0/1585303178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc1940 msgr2=0x55d0cefc6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc1940 0x55d0cefc6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 -- 172.21.15.38:0/1585303178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ced047d0 con 0x55d0cefc0f50 2024-04-22T01:15:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7effdffff640 1 -- 172.21.15.38:0/205679075 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7effec0ced80 con 0x7efff8104b90 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00a2c10 0x7effe00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7efff77fe640 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 0x55d0cefc1370 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7efff0005c40 tx=0x7efff0009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b095881f907e59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.050+0000 7effdd7fa640 1 -- 172.21.15.38:0/1585303178 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff0045070 con 0x55d0cefc0f50 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7effdd7fa640 1 -- 172.21.15.38:0/1585303178 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efff00046c0 con 0x55d0cefc0f50 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7effdd7fa640 1 -- 172.21.15.38:0/1585303178 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff0003850 con 0x55d0cefc0f50 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 -- 172.21.15.38:0/1585303178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 msgr2=0x55d0cefc1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 0x55d0cefc1370 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7efff0005c40 tx=0x7efff0009630 comp rx=0 tx=0).stop 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7efff77fe640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00a2c10 0x7effe00a50d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x55d0cefc9630 tx=0x7efff004c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 -- 172.21.15.38:0/1585303178 shutdown_connections 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefc8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc1940 0x55d0cefc6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 --2- 172.21.15.38:0/1585303178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc0f50 0x55d0cefc1370 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 -- 172.21.15.38:0/1585303178 >> 172.21.15.38:0/1585303178 conn(0x55d0cefbc4c0 msgr2=0x55d0cefbc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 -- 172.21.15.38:0/1585303178 shutdown_connections 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.051+0000 7f000038f740 1 -- 172.21.15.38:0/1585303178 wait complete. 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 -- start start 2024-04-22T01:15:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 Processor -- start 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefcb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefcba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0cefc9a60 con 0x55d0cefc1940 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d0ceee1e00 con 0x55d0cefc6ab0 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0cefc9be0 con 0x55d0cefc0f50 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7f000038f740 10 client.0 _ll_register_callbacks cb 0x55d0cef1ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefcba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefcba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60914/0 (socket says 172.21.15.38:60914) 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 -- 172.21.15.38:0/548510875 learned_addr learned my addr 172.21.15.38:0/548510875 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff77fe640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff6ffd640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefcb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 -- 172.21.15.38:0/548510875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 msgr2=0x55d0cefcb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefcb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 -- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 msgr2=0x55d0cefd2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 -- 172.21.15.38:0/548510875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0cec3a3f0 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff77fe640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7efff67fc640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefcba30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7effec0cf020 tx=0x7effec009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4ca7cac606ca08 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.052+0000 7effdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7effdaff5640 1 -- 172.21.15.38:0/548510875 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec0d0070 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7effdaff5640 1 -- 172.21.15.38:0/548510875 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7effec009cc0 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efff00046c0 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7effdaff5640 1 -- 172.21.15.38:0/548510875 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7effec020070 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 10 client.16476 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d0cebfa0f0 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7effdaff5640 1 -- 172.21.15.38:0/548510875 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efff00046c0 con 0x55d0cefc1940 2024-04-22T01:15:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7effdaff5640 1 -- 172.21.15.38:0/548510875 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7effec0d3040 con 0x55d0cefc1940 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 10 client.16476 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 10 client.16476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 20 client.16476 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cefd7390 0x55d0cefd97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 4 client.16476 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.053+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0cef33aa0 con 0x55d0cefd7390 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.054+0000 7efff6ffd640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cefd7390 0x55d0cefd97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.054+0000 7efff6ffd640 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cefd7390 0x55d0cefd97e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a3f1e6362cd166e server_cookie=39003dbd99aa8f0b in_seq=0 out_seq=0 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.054+0000 7effdaff5640 10 client.16476 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7effdaff5640 1 -- 172.21.15.38:0/548510875 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0cef33aa0 con 0x55d0cefd7390 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7effdaff5640 10 client.16476 handle_command_reply: tid=0 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 client.16476 shutdown 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 msgr2=0x55d0cefcba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefcba30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7effec0cf020 tx=0x7effec009c30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 shutdown_connections 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cefd7390 0x55d0cefd97e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefcba30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7effdaff5640 0 client.16476 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 --2- 172.21.15.38:0/548510875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefcb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 >> 172.21.15.38:0/548510875 conn(0x55d0cefbc4c0 msgr2=0x55d0cefbb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 shutdown_connections 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 1 -- 172.21.15.38:0/548510875 wait complete. 2024-04-22T01:15:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.055+0000 7f000038f740 20 client.16476 trim_cache size 0 max 16384 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.072+0000 7f000038f740 1 Processor -- start 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 -- start start 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc0f50 0x55d0cefbc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefbae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efff81ac610 con 0x55d0cefc1940 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efff81ae150 con 0x55d0cefc0f50 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7f000038f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efff81acce0 con 0x55d0cefc6ab0 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60922/0 (socket says 172.21.15.38:60922) 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 -- 172.21.15.38:0/3551568564 learned_addr learned my addr 172.21.15.38:0/3551568564 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff77fe640 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefbae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff6ffd640 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc0f50 0x55d0cefbc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 -- 172.21.15.38:0/3551568564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 msgr2=0x55d0cefbae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefbae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 -- 172.21.15.38:0/3551568564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc0f50 msgr2=0x55d0cefbc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc0f50 0x55d0cefbc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 -- 172.21.15.38:0/3551568564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0cefc16e0 con 0x55d0cefc1940 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff6ffd640 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc0f50 0x55d0cefbc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7efff67fc640 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd2a30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7effec09b160 tx=0x7effec004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0150e0ecc4ad135 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7effdcff9640 1 -- 172.21.15.38:0/3551568564 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec0d24b0 con 0x55d0cefc1940 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7effdcff9640 1 -- 172.21.15.38:0/3551568564 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effec0d1070 con 0x55d0cefc1940 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.073+0000 7effdcff9640 1 -- 172.21.15.38:0/3551568564 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec09f110 con 0x55d0cefc1940 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 -- 172.21.15.38:0/3551568564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 msgr2=0x55d0cefd2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd2a30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7effec09b160 tx=0x7effec004740 comp rx=0 tx=0).stop 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 -- 172.21.15.38:0/3551568564 shutdown_connections 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc6ab0 0x55d0cefbae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd2a30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 --2- 172.21.15.38:0/3551568564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc0f50 0x55d0cefbc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 -- 172.21.15.38:0/3551568564 >> 172.21.15.38:0/3551568564 conn(0x55d0cefbc4c0 msgr2=0x55d0cefc2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 -- 172.21.15.38:0/3551568564 shutdown_connections 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 -- 172.21.15.38:0/3551568564 wait complete. 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 -- start start 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.074+0000 7f000038f740 1 Processor -- start 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefd0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efff81ada80 con 0x55d0cefc1940 2024-04-22T01:15:53.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efff81ad3b0 con 0x55d0cefc6ab0 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efff81aa5d0 con 0x55d0cefc0f50 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 10 client.0 _ll_register_callbacks cb 0x7effec097560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60930/0 (socket says 172.21.15.38:60930) 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 -- 172.21.15.38:0/3838225323 learned_addr learned my addr 172.21.15.38:0/3838225323 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff77fe640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff6ffd640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefd0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 -- 172.21.15.38:0/3838225323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 msgr2=0x55d0cefd0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefd0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 -- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 msgr2=0x55d0cefd1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 -- 172.21.15.38:0/3838225323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0cefd1b60 con 0x55d0cefc1940 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff77fe640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7efff67fc640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd05b0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7effec00b530 tx=0x7effec009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef37e980c2d102c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7effdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7effdaff5640 1 -- 172.21.15.38:0/3838225323 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec0b9020 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7effdaff5640 1 -- 172.21.15.38:0/3838225323 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7effec0a39b0 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.075+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effec0d1070 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7effdaff5640 1 -- 172.21.15.38:0/3838225323 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7effec0a3e40 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 10 client.16482 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d0cefd2de0 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7effdaff5640 1 -- 172.21.15.38:0/3838225323 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effec0d1070 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7effdaff5640 1 -- 172.21.15.38:0/3838225323 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7effec084190 con 0x55d0cefc1940 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 10 client.16482 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 10 client.16482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 20 client.16482 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cf03de50 0x55d0cf040270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 4 client.16482 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d0cef2ddf0 con 0x55d0cf03de50 2024-04-22T01:15:53.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.076+0000 7efff6ffd640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cf03de50 0x55d0cf040270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7effdaff5640 10 client.16482 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7efff6ffd640 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cf03de50 0x55d0cf040270 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ac4cedf92dc8c5 server_cookie=38efce192bc95d93 in_seq=0 out_seq=0 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7effdaff5640 1 -- 172.21.15.38:0/3838225323 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d0cef2ddf0 con 0x55d0cf03de50 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7effdaff5640 10 client.16482 handle_command_reply: tid=0 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 client.16482 shutdown 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 msgr2=0x55d0cefd05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd05b0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7effec00b530 tx=0x7effec009740 comp rx=0 tx=0).stop 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 shutdown_connections 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x55d0cf03de50 0x55d0cf040270 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d0cefc6ab0 0x55d0cefd1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0cefc1940 0x55d0cefd05b0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7effdaff5640 0 client.16482 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 --2- 172.21.15.38:0/3838225323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0cefc0f50 0x55d0cefd0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 >> 172.21.15.38:0/3838225323 conn(0x55d0cefbc4c0 msgr2=0x55d0cefbf720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 shutdown_connections 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 1 -- 172.21.15.38:0/3838225323 wait complete. 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.077+0000 7f000038f740 20 client.16482 trim_cache size 0 max 16384 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T01:15:53.083 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00a2c10 msgr2=0x7effe00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00a2c10 0x7effe00a50d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x55d0cefc9630 tx=0x7efff004c000 comp rx=0 tx=0).stop 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 msgr2=0x7efff8100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 0x7efff8100210 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7effec00d740 tx=0x7effec00dc00 comp rx=0 tx=0).stop 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 shutdown_connections 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7effe00a2c10 0x7effe00a50d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81074c0 0x7efff8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efff8104b90 0x7efff8100210 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 --2- 172.21.15.38:0/205679075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff8102260 0x7efff8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 >> 172.21.15.38:0/205679075 conn(0x7efff80fbbf0 msgr2=0x7efff8102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 shutdown_connections 2024-04-22T01:15:53.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.078+0000 7efffd6c9640 1 -- 172.21.15.38:0/205679075 wait complete. 2024-04-22T01:15:53.088 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T01:15:53.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:15:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 -- 172.21.15.38:0/3427934191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc103360 msgr2=0x7f06bc109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3427934191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc103360 0x7f06bc109bf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f06a4009900 tx=0x7f06a402bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 -- 172.21.15.38:0/3427934191 shutdown_connections 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3427934191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc103360 0x7f06bc109bf0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3427934191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 0x7f06bc102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3427934191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06bc1017b0 0x7f06bc101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 -- 172.21.15.38:0/3427934191 >> 172.21.15.38:0/3427934191 conn(0x7f06bc0fcd40 msgr2=0x7f06bc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.190+0000 7f06c2b13640 1 -- 172.21.15.38:0/3427934191 shutdown_connections 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 -- 172.21.15.38:0/3427934191 wait complete. 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 Processor -- start 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 -- start start 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 0x7f06bc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 0x7f06bc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06bc103360 0x7f06bc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc10d430 con 0x7f06bc1017b0 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc10d2b0 con 0x7f06bc1029a0 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c2b13640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc10d5b0 con 0x7f06bc103360 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c08af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 0x7f06bc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c08af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 0x7f06bc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60944/0 (socket says 172.21.15.38:60944) 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.191+0000 7f06c08af640 1 -- 172.21.15.38:0/3836826798 learned_addr learned my addr 172.21.15.38:0/3836826798 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c10b0640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06bc103360 0x7f06bc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06bbfff640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 0x7f06bc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c08af640 1 -- 172.21.15.38:0/3836826798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06bc103360 msgr2=0x7f06bc19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c08af640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06bc103360 0x7f06bc19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c08af640 1 -- 172.21.15.38:0/3836826798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 msgr2=0x7f06bc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c08af640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 0x7f06bc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c08af640 1 -- 172.21.15.38:0/3836826798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc19f700 con 0x7f06bc1017b0 2024-04-22T01:15:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06bbfff640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 0x7f06bc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c08af640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 0x7f06bc197ae0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f06b400d950 tx=0x7f06b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4017070 con 0x7f06bc1017b0 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b4009c30 con 0x7f06bc1017b0 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06bc19f9f0 con 0x7f06bc1017b0 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b400f600 con 0x7f06bc1017b0 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.192+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06bc19fef0 con 0x7f06bc1017b0 2024-04-22T01:15:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.193+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06bc108a70 con 0x7f06bc1017b0 2024-04-22T01:15:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.195+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f06b401a070 con 0x7f06bc1017b0 2024-04-22T01:15:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.195+0000 7f06b9ffb640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06a80a2b90 0x7f06a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.195+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f06b40cee50 con 0x7f06bc1017b0 2024-04-22T01:15:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.195+0000 7f06bbfff640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06a80a2b90 0x7f06a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.196+0000 7f06bbfff640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06a80a2b90 0x7f06a80a5050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f06bc198ff0 tx=0x7f06ac009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.199+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f06b4009de0 con 0x7f06bc1017b0 2024-04-22T01:15:53.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.424+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06bc02d170 con 0x7f06bc1017b0 2024-04-22T01:15:53.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.425+0000 7f06b9ffb640 1 -- 172.21.15.38:0/3836826798 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f06b409b9e0 con 0x7f06bc1017b0 2024-04-22T01:15:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:15:53.424 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:15:53.424 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06a80a2b90 msgr2=0x7f06a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06a80a2b90 0x7f06a80a5050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f06bc198ff0 tx=0x7f06ac009520 comp rx=0 tx=0).stop 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 msgr2=0x7f06bc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 0x7f06bc197ae0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f06b400d950 tx=0x7f06b400de10 comp rx=0 tx=0).stop 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 shutdown_connections 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f06a80a2b90 0x7f06a80a5050 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06bc103360 0x7f06bc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06bc1029a0 0x7f06bc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 --2- 172.21.15.38:0/3836826798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06bc1017b0 0x7f06bc197ae0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 >> 172.21.15.38:0/3836826798 conn(0x7f06bc0fcd40 msgr2=0x7f06bc105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 shutdown_connections 2024-04-22T01:15:53.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.427+0000 7f06c2b13640 1 -- 172.21.15.38:0/3836826798 wait complete. 2024-04-22T01:15:53.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T01:15:53.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 -- 172.21.15.38:0/2138764491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 msgr2=0x7f0960104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 --2- 172.21.15.38:0/2138764491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960104670 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f095c009900 tx=0x7f095c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 -- 172.21.15.38:0/2138764491 shutdown_connections 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 --2- 172.21.15.38:0/2138764491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09601074e0 0x7f09601098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 --2- 172.21.15.38:0/2138764491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0960104bb0 0x7f0960106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 --2- 172.21.15.38:0/2138764491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960104670 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 -- 172.21.15.38:0/2138764491 >> 172.21.15.38:0/2138764491 conn(0x7f09600fbc30 msgr2=0x7f09600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 -- 172.21.15.38:0/2138764491 shutdown_connections 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.542+0000 7f0967955640 1 -- 172.21.15.38:0/2138764491 wait complete. 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 Processor -- start 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 -- start start 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0960104bb0 0x7f0960198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09601074e0 0x7f096019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d410 con 0x7f0960102280 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d290 con 0x7f0960104bb0 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0967955640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d590 con 0x7f09601074e0 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60966/0 (socket says 172.21.15.38:60966) 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 -- 172.21.15.38:0/2748306171 learned_addr learned my addr 172.21.15.38:0/2748306171 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0964ef0640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0960104bb0 0x7f0960198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f0965ef2640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09601074e0 0x7f096019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 -- 172.21.15.38:0/2748306171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09601074e0 msgr2=0x7f096019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09601074e0 0x7f096019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 -- 172.21.15.38:0/2748306171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0960104bb0 msgr2=0x7f0960198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0960104bb0 0x7f0960198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.543+0000 7f09656f1640 1 -- 172.21.15.38:0/2748306171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096019f700 con 0x7f0960102280 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.544+0000 7f09656f1640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960197ae0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f095c009230 tx=0x7f095c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.544+0000 7f09567fc640 1 -- 172.21.15.38:0/2748306171 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c041070 con 0x7f0960102280 2024-04-22T01:15:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.544+0000 7f09567fc640 1 -- 172.21.15.38:0/2748306171 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c0054e0 con 0x7f0960102280 2024-04-22T01:15:53.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.544+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096019f990 con 0x7f0960102280 2024-04-22T01:15:53.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.544+0000 7f09567fc640 1 -- 172.21.15.38:0/2748306171 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c034440 con 0x7f0960102280 2024-04-22T01:15:53.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.544+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096019fe80 con 0x7f0960102280 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.546+0000 7f09567fc640 1 -- 172.21.15.38:0/2748306171 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f095c044070 con 0x7f0960102280 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 Processor -- start 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 -- start start 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d1a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1aa30 0x55a525d1f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 0x55a525d22030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a525c7f530 con 0x55a525d1fba0 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a525c27830 con 0x55a525d1aa30 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f096a60e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a525c7b4f0 con 0x55a525d1a040 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f09567fc640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f094c0a2b90 0x7f094c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 0x55a525d22030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0965ef2640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f094c0a2b90 0x7f094c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 0x55a525d22030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60968/0 (socket says 172.21.15.38:60968) 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 -- 172.21.15.38:0/4231126957 learned_addr learned my addr 172.21.15.38:0/4231126957 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f09567fc640 1 -- 172.21.15.38:0/2748306171 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f095c0f11d0 con 0x7f0960102280 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0965ef2640 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d1a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f09656f1640 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1aa30 0x55a525d1f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 -- 172.21.15.38:0/4231126957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 msgr2=0x55a525d1a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d1a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 -- 172.21.15.38:0/4231126957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1aa30 msgr2=0x55a525d1f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1aa30 0x55a525d1f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f0964ef0640 1 -- 172.21.15.38:0/4231126957 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a525c4b2c0 con 0x55a525d1fba0 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.547+0000 7f09656f1640 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1aa30 0x55a525d1f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f0964ef0640 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 0x55a525d22030 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f095000bc40 tx=0x7f095000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebbcb5b7b5a02b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f0965ef2640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f094c0a2b90 0x7f094c0a5050 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55a525d1b290 tx=0x7f0958009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f094b7fe640 1 -- 172.21.15.38:0/4231126957 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950015070 con 0x55a525d1fba0 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f094b7fe640 1 -- 172.21.15.38:0/4231126957 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095001a940 con 0x55a525d1fba0 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f094b7fe640 1 -- 172.21.15.38:0/4231126957 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950011900 con 0x55a525d1fba0 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f096a60e740 1 -- 172.21.15.38:0/4231126957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 msgr2=0x55a525d22030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.548+0000 7f096a60e740 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 0x55a525d22030 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f095000bc40 tx=0x7f095000f730 comp rx=0 tx=0).stop 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- 172.21.15.38:0/4231126957 shutdown_connections 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1fba0 0x55a525d22030 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1aa30 0x55a525d1f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 --2- 172.21.15.38:0/4231126957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d1a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- 172.21.15.38:0/4231126957 >> 172.21.15.38:0/4231126957 conn(0x55a525d155b0 msgr2=0x55a525d159b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- 172.21.15.38:0/4231126957 shutdown_connections 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- 172.21.15.38:0/4231126957 wait complete. 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- start start 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 Processor -- start 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d24220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 0x55a525d24760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a525d22b50 con 0x55a525d1aa30 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a525c3af10 con 0x55a525d1fba0 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a525d22cd0 con 0x55a525d1a040 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f096a60e740 10 client.0 _ll_register_callbacks cb 0x55a525992be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.549+0000 7f0964ef0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 0x55a525d24760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 0x55a525d24760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60980/0 (socket says 172.21.15.38:60980) 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 -- 172.21.15.38:0/602190838 learned_addr learned my addr 172.21.15.38:0/602190838 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0965ef2640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f09656f1640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d24220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 -- 172.21.15.38:0/602190838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 msgr2=0x55a525d24220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d24220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 -- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 msgr2=0x55a525d2b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 -- 172.21.15.38:0/602190838 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a525c0cc80 con 0x55a525d1aa30 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0965ef2640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0964ef0640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 0x55a525d24760 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f095001ad20 tx=0x7f095001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d3a73f8bc51ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0948ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0948ff9640 1 -- 172.21.15.38:0/602190838 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950028060 con 0x55a525d1aa30 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0948ff9640 1 -- 172.21.15.38:0/602190838 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f095001a670 con 0x55a525d1aa30 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095001a940 con 0x55a525d1aa30 2024-04-22T01:15:53.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0948ff9640 1 -- 172.21.15.38:0/602190838 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f095002f050 con 0x55a525d1aa30 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f096a60e740 10 client.16500 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a5259933f0 con 0x55a525d1aa30 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.550+0000 7f0948ff9640 1 -- 172.21.15.38:0/602190838 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095001a940 con 0x55a525d1aa30 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f0948ff9640 1 -- 172.21.15.38:0/602190838 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f09500119f0 con 0x55a525d1aa30 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f096a60e740 10 client.16500 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f096a60e740 10 client.16500 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f096a60e740 20 client.16500 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f096a60e740 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d2fc90 0x55a525d32080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f096a60e740 4 client.16500 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a525c8ca70 con 0x55a525d2fc90 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.551+0000 7f09656f1640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d2fc90 0x55a525d32080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f09656f1640 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d2fc90 0x55a525d32080 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86148005bf72272a server_cookie=2b804740b78b4a10 in_seq=0 out_seq=0 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f0948ff9640 10 client.16500 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f0948ff9640 1 -- 172.21.15.38:0/602190838 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a525c8ca70 con 0x55a525d2fc90 2024-04-22T01:15:53.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f0948ff9640 10 client.16500 handle_command_reply: tid=0 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f096a60e740 1 client.16500 shutdown 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 msgr2=0x55a525d24760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f096a60e740 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 0x55a525d24760 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f095001ad20 tx=0x7f095001ad50 comp rx=0 tx=0).stop 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 shutdown_connections 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f096a60e740 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d2fc90 0x55a525d32080 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.552+0000 7f096a60e740 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f096a60e740 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1aa30 0x55a525d24760 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f0948ff9640 0 client.16500 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f096a60e740 1 --2- 172.21.15.38:0/602190838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1a040 0x55a525d24220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 >> 172.21.15.38:0/602190838 conn(0x55a525d155b0 msgr2=0x55a525d14850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 shutdown_connections 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f096a60e740 1 -- 172.21.15.38:0/602190838 wait complete. 2024-04-22T01:15:53.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.553+0000 7f096a60e740 20 client.16500 trim_cache size 0 max 16384 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 Processor -- start 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 -- start start 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d146d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d2b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d1b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09601aa4f0 con 0x55a525d1a040 2024-04-22T01:15:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f09601ac030 con 0x55a525d1fba0 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f096a60e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09601aabc0 con 0x55a525d1aa30 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d146d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60982/0 (socket says 172.21.15.38:60982) 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 -- 172.21.15.38:0/745215210 learned_addr learned my addr 172.21.15.38:0/745215210 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f0964ef0640 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d2b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 -- 172.21.15.38:0/745215210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 msgr2=0x55a525d2b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d2b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 -- 172.21.15.38:0/745215210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 msgr2=0x55a525d1b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.566+0000 7f09656f1640 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d1b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f09656f1640 1 -- 172.21.15.38:0/745215210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a525a5d7d0 con 0x55a525d1a040 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f09656f1640 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d146d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f095c02cfd0 tx=0x7f095c0a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9d5b263f242f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f094affd640 1 -- 172.21.15.38:0/745215210 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c0d6070 con 0x55a525d1a040 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f094affd640 1 -- 172.21.15.38:0/745215210 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c005690 con 0x55a525d1a040 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f094affd640 1 -- 172.21.15.38:0/745215210 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c04b040 con 0x55a525d1a040 2024-04-22T01:15:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 -- 172.21.15.38:0/745215210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 msgr2=0x55a525d146d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d146d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f095c02cfd0 tx=0x7f095c0a8040 comp rx=0 tx=0).stop 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 -- 172.21.15.38:0/745215210 shutdown_connections 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d1b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d2b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 --2- 172.21.15.38:0/745215210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d146d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 -- 172.21.15.38:0/745215210 >> 172.21.15.38:0/745215210 conn(0x55a525d155b0 msgr2=0x55a525d13fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 -- 172.21.15.38:0/745215210 shutdown_connections 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.567+0000 7f096a60e740 1 -- 172.21.15.38:0/745215210 wait complete. 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 -- start start 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 Processor -- start 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d28ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d29010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09601ab960 con 0x55a525d1a040 2024-04-22T01:15:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f09601ab290 con 0x55a525d1fba0 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09601a84b0 con 0x55a525d1aa30 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f096a60e740 10 client.0 _ll_register_callbacks cb 0x7f0958012650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f09656f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d28ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f09656f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d28ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60988/0 (socket says 172.21.15.38:60988) 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f09656f1640 1 -- 172.21.15.38:0/120777598 learned_addr learned my addr 172.21.15.38:0/120777598 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f0964ef0640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d29010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.568+0000 7f0965ef2640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f09656f1640 1 -- 172.21.15.38:0/120777598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 msgr2=0x55a525d29010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f09656f1640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d29010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f09656f1640 1 -- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 msgr2=0x55a525d2a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f09656f1640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f09656f1640 1 -- 172.21.15.38:0/120777598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a525d2a610 con 0x55a525d1a040 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f09656f1640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d28ad0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f095c0c5da0 tx=0x7f095c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40f668badf070d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f0948ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f0948ff9640 1 -- 172.21.15.38:0/120777598 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c04b070 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f0948ff9640 1 -- 172.21.15.38:0/120777598 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f095c0b9040 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a525c86dc0 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f0948ff9640 1 -- 172.21.15.38:0/120777598 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f095c0d6040 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 10 client.16506 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a525d2b910 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f0948ff9640 1 -- 172.21.15.38:0/120777598 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a525c86dc0 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f0948ff9640 1 -- 172.21.15.38:0/120777598 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f095c084c70 con 0x55a525d1a040 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 10 client.16506 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 10 client.16506 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 20 client.16506 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d96680 0x55a525d98aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 4 client.16506 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.569+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 --> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a525c046e0 con 0x55a525d96680 2024-04-22T01:15:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f0964ef0640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d96680 0x55a525d98aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f0948ff9640 10 client.16506 ms_handle_connect on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f0964ef0640 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d96680 0x55a525d98aa0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=583377bbd528381 server_cookie=f04d8a13a415acc3 in_seq=0 out_seq=0 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f0948ff9640 1 -- 172.21.15.38:0/120777598 <== mds.4 v2:172.21.15.111:6834/2761116669 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a525c046e0 con 0x55a525d96680 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f0948ff9640 10 client.16506 handle_command_reply: tid=0 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 client.16506 shutdown 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 msgr2=0x55a525d28ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d28ad0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f095c0c5da0 tx=0x7f095c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 shutdown_connections 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:6834/2761116669,v1:172.21.15.111:6835/2761116669] conn(0x55a525d96680 0x55a525d98aa0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a525d1fba0 0x55a525d2a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f0948ff9640 0 client.16506 ms_handle_reset on v2:172.21.15.111:6834/2761116669 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a525d1aa30 0x55a525d29010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 --2- 172.21.15.38:0/120777598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a525d1a040 0x55a525d28ad0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.570+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 >> 172.21.15.38:0/120777598 conn(0x55a525d155b0 msgr2=0x55a525d190a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 shutdown_connections 2024-04-22T01:15:53.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f096a60e740 1 -- 172.21.15.38:0/120777598 wait complete. 2024-04-22T01:15:53.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f096a60e740 20 client.16506 trim_cache size 0 max 16384 2024-04-22T01:15:53.576 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f094c0a2b90 msgr2=0x7f094c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f094c0a2b90 0x7f094c0a5050 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55a525d1b290 tx=0x7f0958009550 comp rx=0 tx=0).stop 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 msgr2=0x7f0960197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960197ae0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f095c009230 tx=0x7f095c005dd0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 shutdown_connections 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f094c0a2b90 0x7f094c0a5050 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09601074e0 0x7f096019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0960104bb0 0x7f0960198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 --2- 172.21.15.38:0/2748306171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0960102280 0x7f0960197ae0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 >> 172.21.15.38:0/2748306171 conn(0x7f09600fbc30 msgr2=0x7f0960102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 shutdown_connections 2024-04-22T01:15:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.571+0000 7f0967955640 1 -- 172.21.15.38:0/2748306171 wait complete. 2024-04-22T01:15:53.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T01:15:53.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 -- 172.21.15.38:0/1203581397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458100680 msgr2=0x7f7458100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 --2- 172.21.15.38:0/1203581397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458100680 0x7f7458100aa0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7448009900 tx=0x7f744802bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 -- 172.21.15.38:0/1203581397 shutdown_connections 2024-04-22T01:15:53.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 --2- 172.21.15.38:0/1203581397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458102230 0x7f7458108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 --2- 172.21.15.38:0/1203581397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458101870 0x7f7458101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 --2- 172.21.15.38:0/1203581397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458100680 0x7f7458100aa0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 -- 172.21.15.38:0/1203581397 >> 172.21.15.38:0/1203581397 conn(0x7f74580fbbf0 msgr2=0x7f74580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 -- 172.21.15.38:0/1203581397 shutdown_connections 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 -- 172.21.15.38:0/1203581397 wait complete. 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.686+0000 7f745d695640 1 Processor -- start 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 -- start start 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458100680 0x7f7458197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 0x7f7458198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458102230 0x7f745819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f745810d420 con 0x7f7458101870 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f745810d2a0 con 0x7f7458100680 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f745d695640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f745810d5a0 con 0x7f7458102230 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 0x7f7458198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 0x7f7458198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32772/0 (socket says 172.21.15.38:32772) 2024-04-22T01:15:53.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 -- 172.21.15.38:0/3772962469 learned_addr learned my addr 172.21.15.38:0/3772962469 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74577fe640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458102230 0x7f745819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f7456ffd640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458100680 0x7f7458197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 -- 172.21.15.38:0/3772962469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458102230 msgr2=0x7f745819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458102230 0x7f745819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 -- 172.21.15.38:0/3772962469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458100680 msgr2=0x7f7458197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458100680 0x7f7458197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 -- 172.21.15.38:0/3772962469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745819f700 con 0x7f7458101870 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f7456ffd640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458100680 0x7f7458197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f74567fc640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 0x7f7458198020 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f744c00d950 tx=0x7f744c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.687+0000 7f743ffff640 1 -- 172.21.15.38:0/3772962469 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c017030 con 0x7f7458101870 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.688+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f745819f9f0 con 0x7f7458101870 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.688+0000 7f743ffff640 1 -- 172.21.15.38:0/3772962469 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f744c004020 con 0x7f7458101870 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.688+0000 7f743ffff640 1 -- 172.21.15.38:0/3772962469 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c0047a0 con 0x7f7458101870 2024-04-22T01:15:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.688+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f745819ff90 con 0x7f7458101870 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.690+0000 7f743ffff640 1 -- 172.21.15.38:0/3772962469 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f744c0041d0 con 0x7f7458101870 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 Processor -- start 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 -- start start 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3450 0x5609679f3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x5609679f8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x5609679fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560967954530 con 0x5609679f3e40 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5609678fc830 con 0x5609679f3450 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f746027d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5609679504f0 con 0x5609679f8fb0 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x5609679f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x5609679f8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32788/0 (socket says 172.21.15.38:32788) 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 -- 172.21.15.38:0/3884844308 learned_addr learned my addr 172.21.15.38:0/3884844308 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f743ffff640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74400a2b90 0x7f74400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f74577fe640 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3450 0x5609679f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f743ffff640 1 -- 172.21.15.38:0/3772962469 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f744c0ceee0 con 0x7f7458101870 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f74567fc640 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x5609679fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f74577fe640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74400a2b90 0x7f74400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 -- 172.21.15.38:0/3884844308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 msgr2=0x5609679fb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x5609679fb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 -- 172.21.15.38:0/3884844308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3450 msgr2=0x5609679f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3450 0x5609679f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f7456ffd640 1 -- 172.21.15.38:0/3884844308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609679202c0 con 0x5609679f3e40 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.691+0000 7f74567fc640 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x5609679fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f7456ffd640 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x5609679f8940 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7448006010 tx=0x7f7448005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e4344c62bcff12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f74577fe640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74400a2b90 0x7f74400a5050 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x5609679f46a0 tx=0x7f7450009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f743d7fa640 1 -- 172.21.15.38:0/3884844308 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744802ccc0 con 0x5609679f3e40 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f743d7fa640 1 -- 172.21.15.38:0/3884844308 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7448005040 con 0x5609679f3e40 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f743d7fa640 1 -- 172.21.15.38:0/3884844308 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7448004a20 con 0x5609679f3e40 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 -- 172.21.15.38:0/3884844308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 msgr2=0x5609679f8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x5609679f8940 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7448006010 tx=0x7f7448005d50 comp rx=0 tx=0).stop 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 -- 172.21.15.38:0/3884844308 shutdown_connections 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x5609679fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x5609679f8940 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 --2- 172.21.15.38:0/3884844308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3450 0x5609679f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.692+0000 7f746027d740 1 -- 172.21.15.38:0/3884844308 >> 172.21.15.38:0/3884844308 conn(0x5609679ea5b0 msgr2=0x5609679ea9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 -- 172.21.15.38:0/3884844308 shutdown_connections 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 -- 172.21.15.38:0/3884844308 wait complete. 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 -- start start 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 Processor -- start 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 0x5609679fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3e40 0x5609679fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x560967a04b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5609679fbf00 con 0x5609679f3450 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56096790ff10 con 0x5609679f3e40 2024-04-22T01:15:53.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5609679fc080 con 0x5609679f8fb0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f746027d740 10 client.0 _ll_register_callbacks cb 0x560967667be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f7456ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 0x5609679fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f7456ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 0x5609679fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32796/0 (socket says 172.21.15.38:32796) 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f7456ffd640 1 -- 172.21.15.38:0/4153653732 learned_addr learned my addr 172.21.15.38:0/4153653732 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f74577fe640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x560967a04b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f7456ffd640 1 -- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 msgr2=0x560967a04b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f7456ffd640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x560967a04b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f7456ffd640 1 -- 172.21.15.38:0/4153653732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3e40 msgr2=0x5609679fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.693+0000 7f74567fc640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3e40 0x5609679fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f7456ffd640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3e40 0x5609679fdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f7456ffd640 1 -- 172.21.15.38:0/4153653732 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609678e1c80 con 0x5609679f3450 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f74577fe640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x560967a04b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f7456ffd640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 0x5609679fd600 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f744802c7b0 tx=0x7f744802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09a8f0983e1e72d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f743aff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f743aff5640 1 -- 172.21.15.38:0/4153653732 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744804c020 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f743aff5640 1 -- 172.21.15.38:0/4153653732 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f74480047a0 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7448005040 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f743aff5640 1 -- 172.21.15.38:0/4153653732 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7448057050 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f746027d740 10 client.16518 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609676683f0 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f743aff5640 1 -- 172.21.15.38:0/4153653732 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7448005040 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f743aff5640 1 -- 172.21.15.38:0/4153653732 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7448003b90 con 0x5609679f3450 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f746027d740 10 client.16518 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.694+0000 7f746027d740 10 client.16518 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f746027d740 20 client.16518 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f746027d740 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a08f70 0x560967a0b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f746027d740 4 client.16518 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560967961a70 con 0x560967a08f70 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f74567fc640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a08f70 0x560967a0b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f74567fc640 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a08f70 0x560967a0b360 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4394f9bd44dd338d server_cookie=96ab59733c505bfb in_seq=0 out_seq=0 2024-04-22T01:15:53.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.695+0000 7f743aff5640 10 client.16518 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f743aff5640 1 -- 172.21.15.38:0/4153653732 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560967961a70 con 0x560967a08f70 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f743aff5640 10 client.16518 handle_command_reply: tid=0 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 client.16518 shutdown 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 msgr2=0x5609679fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 0x5609679fd600 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f744802c7b0 tx=0x7f744802bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 shutdown_connections 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a08f70 0x560967a0b360 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f8fb0 0x560967a04b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f743aff5640 0 client.16518 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f3e40 0x5609679fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 --2- 172.21.15.38:0/4153653732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3450 0x5609679fd600 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 >> 172.21.15.38:0/4153653732 conn(0x5609679ea5b0 msgr2=0x5609679e9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 shutdown_connections 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 1 -- 172.21.15.38:0/4153653732 wait complete. 2024-04-22T01:15:53.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.696+0000 7f746027d740 20 client.16518 trim_cache size 0 max 16384 2024-04-22T01:15:53.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 Processor -- start 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 -- start start 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 0x5609679e9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a04ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a10be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74581a8300 con 0x5609679f3e40 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f74581a9e40 con 0x5609679f8fb0 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f746027d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74581a89d0 con 0x5609679f3450 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a04ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a04ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32810/0 (socket says 172.21.15.38:32810) 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 -- 172.21.15.38:0/2933645187 learned_addr learned my addr 172.21.15.38:0/2933645187 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 -- 172.21.15.38:0/2933645187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 msgr2=0x5609679e9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 0x5609679e9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74577fe640 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a10be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 -- 172.21.15.38:0/2933645187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 msgr2=0x560967a10be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a10be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 -- 172.21.15.38:0/2933645187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609677327d0 con 0x5609679f3e40 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74577fe640 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a10be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.710+0000 7f74567fc640 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a04ad0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f744c097790 tx=0x7f744c0bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53881bbacbb6b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f743cff9640 1 -- 172.21.15.38:0/2933645187 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c09f6e0 con 0x5609679f3e40 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f743cff9640 1 -- 172.21.15.38:0/2933645187 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f744c0a3fa0 con 0x5609679f3e40 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f743cff9640 1 -- 172.21.15.38:0/2933645187 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c09a9e0 con 0x5609679f3e40 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 -- 172.21.15.38:0/2933645187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 msgr2=0x560967a04ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a04ad0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f744c097790 tx=0x7f744c0bb040 comp rx=0 tx=0).stop 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 -- 172.21.15.38:0/2933645187 shutdown_connections 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a10be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a04ad0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 --2- 172.21.15.38:0/2933645187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 0x5609679e9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 -- 172.21.15.38:0/2933645187 >> 172.21.15.38:0/2933645187 conn(0x5609679ea5b0 msgr2=0x5609679f5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 -- 172.21.15.38:0/2933645187 shutdown_connections 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 -- 172.21.15.38:0/2933645187 wait complete. 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 -- start start 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.711+0000 7f746027d740 1 Processor -- start 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 0x560967a023a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a028e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a038f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74581a9770 con 0x5609679f3e40 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f74581a90a0 con 0x5609679f8fb0 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74581a62c0 con 0x5609679f3450 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 10 client.0 _ll_register_callbacks cb 0x7f7448009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a028e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32826/0 (socket says 172.21.15.38:32826) 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 -- 172.21.15.38:0/1517459654 learned_addr learned my addr 172.21.15.38:0/1517459654 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74577fe640 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 -- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 msgr2=0x560967a023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 0x560967a023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 -- 172.21.15.38:0/1517459654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 msgr2=0x560967a038f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a038f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 -- 172.21.15.38:0/1517459654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560967a03ee0 con 0x5609679f3e40 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f74567fc640 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a028e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f744c005fd0 tx=0x7f744c00df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb8a175dc007c76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f743aff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f743aff5640 1 -- 172.21.15.38:0/1517459654 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c0b4200 con 0x5609679f3e40 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f743aff5640 1 -- 172.21.15.38:0/1517459654 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f744c09a780 con 0x5609679f3e40 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f744c0a3fa0 con 0x5609679f3e40 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f743aff5640 1 -- 172.21.15.38:0/1517459654 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f744c09a980 con 0x5609679f3e40 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 10 client.16524 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.712+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560967a6f5b0 con 0x5609679f3e40 2024-04-22T01:15:53.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f743aff5640 1 -- 172.21.15.38:0/1517459654 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f744c0a3fa0 con 0x5609679f3e40 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f743aff5640 1 -- 172.21.15.38:0/1517459654 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f744c0971a0 con 0x5609679f3e40 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f746027d740 10 client.16524 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f746027d740 10 client.16524 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f746027d740 20 client.16524 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f746027d740 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a6ff20 0x560967a72340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f746027d740 4 client.16524 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 --> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609679ee670 con 0x560967a6ff20 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f7456ffd640 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a6ff20 0x560967a72340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f743aff5640 10 client.16524 ms_handle_connect on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.713+0000 7f7456ffd640 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a6ff20 0x560967a72340 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57e548c2bc843220 server_cookie=bc9d1169e553154d in_seq=0 out_seq=0 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f743aff5640 1 -- 172.21.15.38:0/1517459654 <== mds.3 v2:172.21.15.203:6836/2993816469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609679ee670 con 0x560967a6ff20 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f743aff5640 10 client.16524 handle_command_reply: tid=0 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 client.16524 shutdown 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 msgr2=0x560967a028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a028e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f744c005fd0 tx=0x7f744c00df40 comp rx=0 tx=0).stop 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 shutdown_connections 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:6836/2993816469,v1:172.21.15.203:6837/2993816469] conn(0x560967a6ff20 0x560967a72340 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5609679f8fb0 0x560967a038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5609679f3e40 0x560967a028e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 --2- 172.21.15.38:0/1517459654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5609679f3450 0x560967a023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f743aff5640 0 client.16524 ms_handle_reset on v2:172.21.15.203:6836/2993816469 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 >> 172.21.15.38:0/1517459654 conn(0x5609679ea5b0 msgr2=0x560967a0fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 shutdown_connections 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 1 -- 172.21.15.38:0/1517459654 wait complete. 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f746027d740 20 client.16524 trim_cache size 0 max 16384 2024-04-22T01:15:53.719 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:53.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74400a2b90 msgr2=0x7f74400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f745d695640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74400a2b90 0x7f74400a5050 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x5609679f46a0 tx=0x7f7450009600 comp rx=0 tx=0).stop 2024-04-22T01:15:53.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 msgr2=0x7f7458198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.714+0000 7f745d695640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 0x7f7458198020 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f744c00d950 tx=0x7f744c00de10 comp rx=0 tx=0).stop 2024-04-22T01:15:53.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 shutdown_connections 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f74400a2b90 0x7f74400a5050 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458102230 0x7f745819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7458101870 0x7f7458198020 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 --2- 172.21.15.38:0/3772962469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7458100680 0x7f7458197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 >> 172.21.15.38:0/3772962469 conn(0x7f74580fbbf0 msgr2=0x7f7458104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 shutdown_connections 2024-04-22T01:15:53.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.715+0000 7f745d695640 1 -- 172.21.15.38:0/3772962469 wait complete. 2024-04-22T01:15:53.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T01:15:53.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 -- 172.21.15.38:0/3076855790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa034102250 msgr2=0x7fa034108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 --2- 172.21.15.38:0/3076855790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa034102250 0x7fa034108ae0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009ca0 tx=0x7fa02c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 -- 172.21.15.38:0/3076855790 shutdown_connections 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 --2- 172.21.15.38:0/3076855790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa034102250 0x7fa034108ae0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 --2- 172.21.15.38:0/3076855790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa034101890 0x7fa034101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 --2- 172.21.15.38:0/3076855790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0341006a0 0x7fa034100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 -- 172.21.15.38:0/3076855790 >> 172.21.15.38:0/3076855790 conn(0x7fa0340fbc30 msgr2=0x7fa0340fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 -- 172.21.15.38:0/3076855790 shutdown_connections 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 -- 172.21.15.38:0/3076855790 wait complete. 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.832+0000 7fa03ab02640 1 Processor -- start 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 -- start start 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 0x7fa03406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa034101890 0x7fa03419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa034102250 0x7fa03419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa03410c320 con 0x7fa0341006a0 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa03410c1a0 con 0x7fa034101890 2024-04-22T01:15:53.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03ab02640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa03410c4a0 con 0x7fa034102250 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 0x7fa03406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 0x7fa03406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32834/0 (socket says 172.21.15.38:32834) 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 -- 172.21.15.38:0/513130388 learned_addr learned my addr 172.21.15.38:0/513130388 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03909f640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa034102250 0x7fa03419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa033fff640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa034101890 0x7fa03419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 -- 172.21.15.38:0/513130388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa034102250 msgr2=0x7fa03419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa034102250 0x7fa03419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 -- 172.21.15.38:0/513130388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa034101890 msgr2=0x7fa03419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa034101890 0x7fa03419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 -- 172.21.15.38:0/513130388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03419f6c0 con 0x7fa0341006a0 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03909f640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa034102250 0x7fa03419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa03889e640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 0x7fa03406a8d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa02800d950 tx=0x7fa02800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.833+0000 7fa031ffb640 1 -- 172.21.15.38:0/513130388 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028017070 con 0x7fa0341006a0 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.834+0000 7fa031ffb640 1 -- 172.21.15.38:0/513130388 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa028009c30 con 0x7fa0341006a0 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.834+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03419f9b0 con 0x7fa0341006a0 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.834+0000 7fa031ffb640 1 -- 172.21.15.38:0/513130388 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02800f600 con 0x7fa0341006a0 2024-04-22T01:15:53.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.834+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03419feb0 con 0x7fa0341006a0 2024-04-22T01:15:53.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa031ffb640 1 -- 172.21.15.38:0/513130388 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa02801a070 con 0x7fa0341006a0 2024-04-22T01:15:53.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 Processor -- start 2024-04-22T01:15:53.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 -- start start 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5564d392b530 con 0x5564d39c6040 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5564d38d3830 con 0x5564d39c6a30 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564d39274f0 con 0x5564d39cbba0 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03909f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03909f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32848/0 (socket says 172.21.15.38:32848) 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.836+0000 7fa03909f640 1 -- 172.21.15.38:0/2069764127 learned_addr learned my addr 172.21.15.38:0/2069764127 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa031ffb640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa0240a2b90 0x7fa0240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa033fff640 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 -- 172.21.15.38:0/2069764127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 msgr2=0x5564d39ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 -- 172.21.15.38:0/2069764127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 msgr2=0x5564d39cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa031ffb640 1 -- 172.21.15.38:0/513130388 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa0280cef60 con 0x7fa0341006a0 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03889e640 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 -- 172.21.15.38:0/2069764127 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d38f72c0 con 0x5564d39c6040 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03889e640 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa0240a2b90 0x7fa0240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa03909f640 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c6460 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0324c0 tx=0x7fa02c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db42fa9aa79d334 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa01effd640 1 -- 172.21.15.38:0/2069764127 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c039040 con 0x5564d39c6040 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa01effd640 1 -- 172.21.15.38:0/2069764127 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c003b70 con 0x5564d39c6040 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.837+0000 7fa01effd640 1 -- 172.21.15.38:0/2069764127 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0346a0 con 0x5564d39c6040 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 -- 172.21.15.38:0/2069764127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 msgr2=0x5564d39c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c6460 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0324c0 tx=0x7fa02c009880 comp rx=0 tx=0).stop 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03909f640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa0240a2b90 0x7fa0240a5050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa02c047000 tx=0x7fa02c044000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 -- 172.21.15.38:0/2069764127 shutdown_connections 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/2069764127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c6460 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 -- 172.21.15.38:0/2069764127 >> 172.21.15.38:0/2069764127 conn(0x5564d39c15b0 msgr2=0x5564d39c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 -- 172.21.15.38:0/2069764127 shutdown_connections 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 -- 172.21.15.38:0/2069764127 wait complete. 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 -- start start 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.838+0000 7fa03d7c8740 1 Processor -- start 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5564d39ceb50 con 0x5564d39c6a30 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5564d38e6f10 con 0x5564d39cbba0 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564d39cecd0 con 0x5564d39c6040 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03d7c8740 10 client.0 _ll_register_callbacks cb 0x5564d363ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32860/0 (socket says 172.21.15.38:32860) 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 -- 172.21.15.38:0/1156089295 learned_addr learned my addr 172.21.15.38:0/1156089295 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03889e640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 -- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 msgr2=0x5564d39d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 -- 172.21.15.38:0/1156089295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 msgr2=0x5564d39d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03909f640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa033fff640 1 -- 172.21.15.38:0/1156089295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d38b8c80 con 0x5564d39c6a30 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03889e640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.839+0000 7fa03909f640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa033fff640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d0770 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa02000dee0 tx=0x7fa0200117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3b2225779720a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa013fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa013fff640 1 -- 172.21.15.38:0/1156089295 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020004220 con 0x5564d39c6a30 2024-04-22T01:15:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02c003b70 con 0x5564d39c6a30 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa013fff640 1 -- 172.21.15.38:0/1156089295 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa0200043c0 con 0x5564d39c6a30 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa013fff640 1 -- 172.21.15.38:0/1156089295 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa020024050 con 0x5564d39c6a30 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 10 client.16536 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564d363f3f0 con 0x5564d39c6a30 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa013fff640 1 -- 172.21.15.38:0/1156089295 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c003b70 con 0x5564d39c6a30 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa013fff640 1 -- 172.21.15.38:0/1156089295 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa02001b880 con 0x5564d39c6a30 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 10 client.16536 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 10 client.16536 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 20 client.16536 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d39dc090 0x5564d39de550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 4 client.16536 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.840+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564d3938a70 con 0x5564d39dc090 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.841+0000 7fa03889e640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d39dc090 0x5564d39de550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.841+0000 7fa013fff640 10 client.16536 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:53.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.841+0000 7fa03889e640 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d39dc090 0x5564d39de550 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=204819c7ba092e94 server_cookie=222ff4c8ec2001d2 in_seq=0 out_seq=0 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa013fff640 1 -- 172.21.15.38:0/1156089295 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564d3938a70 con 0x5564d39dc090 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa013fff640 10 client.16536 handle_command_reply: tid=0 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 client.16536 shutdown 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 msgr2=0x5564d39d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d0770 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa02000dee0 tx=0x7fa0200117b0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 shutdown_connections 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d39dc090 0x5564d39de550 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa013fff640 0 client.16536 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d0770 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1156089295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 >> 172.21.15.38:0/1156089295 conn(0x5564d39c15b0 msgr2=0x5564d39c0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 shutdown_connections 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1156089295 wait complete. 2024-04-22T01:15:53.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.842+0000 7fa03d7c8740 20 client.16536 trim_cache size 0 max 16384 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.855+0000 7fa03d7c8740 1 Processor -- start 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.855+0000 7fa03d7c8740 1 -- start start 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39d7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa0341aa500 con 0x5564d39c6040 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa0341ac040 con 0x5564d39c6a30 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0341aabd0 con 0x5564d39cbba0 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32876/0 (socket says 172.21.15.38:32876) 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 -- 172.21.15.38:0/3653024058 learned_addr learned my addr 172.21.15.38:0/3653024058 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03909f640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa033fff640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 -- 172.21.15.38:0/3653024058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 msgr2=0x5564d39e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39e2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 -- 172.21.15.38:0/3653024058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 msgr2=0x5564d39d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39d7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 -- 172.21.15.38:0/3653024058 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d37097d0 con 0x5564d39c6040 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa033fff640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03909f640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa03889e640 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c06d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa028009610 tx=0x7fa028097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0d61cf28eef927 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa01e7fc640 1 -- 172.21.15.38:0/3653024058 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0280b4070 con 0x5564d39c6040 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa01e7fc640 1 -- 172.21.15.38:0/3653024058 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0280a4060 con 0x5564d39c6040 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.856+0000 7fa01e7fc640 1 -- 172.21.15.38:0/3653024058 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028021040 con 0x5564d39c6040 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 -- 172.21.15.38:0/3653024058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 msgr2=0x5564d39c06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c06d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa028009610 tx=0x7fa028097040 comp rx=0 tx=0).stop 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 -- 172.21.15.38:0/3653024058 shutdown_connections 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39cbba0 0x5564d39e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39c6a30 0x5564d39d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/3653024058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6040 0x5564d39c06d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 -- 172.21.15.38:0/3653024058 >> 172.21.15.38:0/3653024058 conn(0x5564d39c15b0 msgr2=0x5564d39bffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 -- 172.21.15.38:0/3653024058 shutdown_connections 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 -- 172.21.15.38:0/3653024058 wait complete. 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 -- start start 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.857+0000 7fa03d7c8740 1 Processor -- start 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa0341ab970 con 0x5564d39c6a30 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa0341ab2a0 con 0x5564d39cbba0 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0341a84c0 con 0x5564d39c6040 2024-04-22T01:15:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 10 client.0 _ll_register_callbacks cb 0x7fa02000d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32890/0 (socket says 172.21.15.38:32890) 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 -- 172.21.15.38:0/1828541874 learned_addr learned my addr 172.21.15.38:0/1828541874 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03909f640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03889e640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 -- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 msgr2=0x5564d39d55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 -- 172.21.15.38:0/1828541874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 msgr2=0x5564d39d6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 -- 172.21.15.38:0/1828541874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d39d7100 con 0x5564d39c6a30 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03889e640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa033fff640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d5b00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa020006010 tx=0x7fa02000d670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7235ff17a52680b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa013fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa013fff640 1 -- 172.21.15.38:0/1828541874 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020021040 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0280a4060 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.858+0000 7fa013fff640 1 -- 172.21.15.38:0/1828541874 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa020004220 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa013fff640 1 -- 172.21.15.38:0/1828541874 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa020004460 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 10 client.16542 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564d3a42da0 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa013fff640 1 -- 172.21.15.38:0/1828541874 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0280a4060 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa013fff640 1 -- 172.21.15.38:0/1828541874 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa020004680 con 0x5564d39c6a30 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 10 client.16542 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 10 client.16542 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 20 client.16542 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d3a437f0 0x5564d3a45c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 4 client.16542 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 --> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564d3932dc0 con 0x5564d3a437f0 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.859+0000 7fa03889e640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d3a437f0 0x5564d3a45c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03889e640 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d3a437f0 0x5564d3a45c10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59d9d515f1033d1f server_cookie=18fd671e6504c59a in_seq=0 out_seq=0 2024-04-22T01:15:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa013fff640 10 client.16542 ms_handle_connect on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa013fff640 1 -- 172.21.15.38:0/1828541874 <== mds.2 v2:172.21.15.203:6838/765839756 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564d3932dc0 con 0x5564d3a437f0 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa013fff640 10 client.16542 handle_command_reply: tid=0 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 client.16542 shutdown 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 msgr2=0x5564d39d5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d5b00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa020006010 tx=0x7fa02000d670 comp rx=0 tx=0).stop 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 shutdown_connections 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:6838/765839756,v1:172.21.15.203:6839/765839756] conn(0x5564d3a437f0 0x5564d3a45c10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5564d39cbba0 0x5564d39d6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5564d39c6a30 0x5564d39d5b00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 --2- 172.21.15.38:0/1828541874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564d39c6040 0x5564d39d55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa013fff640 0 client.16542 ms_handle_reset on v2:172.21.15.203:6838/765839756 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 >> 172.21.15.38:0/1828541874 conn(0x5564d39c15b0 msgr2=0x5564d39c0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 shutdown_connections 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 1 -- 172.21.15.38:0/1828541874 wait complete. 2024-04-22T01:15:53.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.860+0000 7fa03d7c8740 20 client.16542 trim_cache size 0 max 16384 2024-04-22T01:15:53.867 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:53.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa0240a2b90 msgr2=0x7fa0240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa0240a2b90 0x7fa0240a5050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa02c047000 tx=0x7fa02c044000 comp rx=0 tx=0).stop 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 msgr2=0x7fa03406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 0x7fa03406a8d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa02800d950 tx=0x7fa02800de10 comp rx=0 tx=0).stop 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 shutdown_connections 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa0240a2b90 0x7fa0240a5050 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa034102250 0x7fa03419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa034101890 0x7fa03419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 --2- 172.21.15.38:0/513130388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0341006a0 0x7fa03406a8d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 >> 172.21.15.38:0/513130388 conn(0x7fa0340fbc30 msgr2=0x7fa034104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 shutdown_connections 2024-04-22T01:15:53.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.861+0000 7fa03ab02640 1 -- 172.21.15.38:0/513130388 wait complete. 2024-04-22T01:15:53.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 -- 172.21.15.38:0/3040581597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65781074e0 msgr2=0x7f65781098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 --2- 172.21.15.38:0/3040581597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65781074e0 0x7f65781098d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6568009ca0 tx=0x7f656802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 -- 172.21.15.38:0/3040581597 shutdown_connections 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 --2- 172.21.15.38:0/3040581597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65781074e0 0x7f65781098d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 --2- 172.21.15.38:0/3040581597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6578104bb0 0x7f6578106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 --2- 172.21.15.38:0/3040581597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6578102280 0x7f6578104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 -- 172.21.15.38:0/3040581597 >> 172.21.15.38:0/3040581597 conn(0x7f65780fbc30 msgr2=0x7f65780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 -- 172.21.15.38:0/3040581597 shutdown_connections 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.976+0000 7f657d56c640 1 -- 172.21.15.38:0/3040581597 wait complete. 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 Processor -- start 2024-04-22T01:15:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 -- start start 2024-04-22T01:15:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 0x7f6578197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6578104bb0 0x7f6578197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65781074e0 0x7f657819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f657810d440 con 0x7f6578102280 2024-04-22T01:15:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f657810d2c0 con 0x7f65781074e0 2024-04-22T01:15:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f657d56c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f657810d5c0 con 0x7f6578104bb0 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f6576ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 0x7f6578197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f6576ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 0x7f6578197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32912/0 (socket says 172.21.15.38:32912) 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f6576ffd640 1 -- 172.21.15.38:0/1611177601 learned_addr learned my addr 172.21.15.38:0/1611177601 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.977+0000 7f65777fe640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65781074e0 0x7f657819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f65767fc640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6578104bb0 0x7f6578197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f6576ffd640 1 -- 172.21.15.38:0/1611177601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6578104bb0 msgr2=0x7f6578197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f6576ffd640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6578104bb0 0x7f6578197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f6576ffd640 1 -- 172.21.15.38:0/1611177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65781074e0 msgr2=0x7f657819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f6576ffd640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65781074e0 0x7f657819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f6576ffd640 1 -- 172.21.15.38:0/1611177601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657819f6a0 con 0x7f6578102280 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f6576ffd640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 0x7f6578197a80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f657000e9f0 tx=0x7f657000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f655ffff640 1 -- 172.21.15.38:0/1611177601 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657000ccc0 con 0x7f6578102280 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657819f990 con 0x7f6578102280 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f655ffff640 1 -- 172.21.15.38:0/1611177601 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f657000ce60 con 0x7f6578102280 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f655ffff640 1 -- 172.21.15.38:0/1611177601 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570010690 con 0x7f6578102280 2024-04-22T01:15:53.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.978+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657819fe90 con 0x7f6578102280 2024-04-22T01:15:53.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.980+0000 7f655ffff640 1 -- 172.21.15.38:0/1611177601 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f65700046c0 con 0x7f6578102280 2024-04-22T01:15:53.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 Processor -- start 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 -- start start 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 0x55a9440a4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4a30 0x55a9440a9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440ac030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a944009530 con 0x55a9440a4040 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a943fb1830 con 0x55a9440a4a30 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6580232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a9440054f0 con 0x55a9440a9ba0 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f655ffff640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f65600a2b90 0x7f65600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 0x55a9440a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f655ffff640 1 -- 172.21.15.38:0/1611177601 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f65700cf830 con 0x7f6578102280 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 0x55a9440a4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32926/0 (socket says 172.21.15.38:32926) 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 -- 172.21.15.38:0/3961850518 learned_addr learned my addr 172.21.15.38:0/3961850518 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f65600a2b90 0x7f65600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65767fc640 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f6576ffd640 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4a30 0x55a9440a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 -- 172.21.15.38:0/3961850518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 msgr2=0x55a9440ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.981+0000 7f65777fe640 1 -- 172.21.15.38:0/3961850518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4a30 msgr2=0x55a9440a9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f65777fe640 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4a30 0x55a9440a9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f65777fe640 1 -- 172.21.15.38:0/3961850518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a943fd52c0 con 0x55a9440a4040 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f65777fe640 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 0x55a9440a4460 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6568009ca0 tx=0x7f6568032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f55979b9b536b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f65777fe640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f65600a2b90 0x7f65600a5050 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6568036000 tx=0x7f6568042000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f655d7fa640 1 -- 172.21.15.38:0/3961850518 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568051070 con 0x55a9440a4040 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f655d7fa640 1 -- 172.21.15.38:0/3961850518 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6568004b40 con 0x55a9440a4040 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.982+0000 7f655d7fa640 1 -- 172.21.15.38:0/3961850518 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568039040 con 0x55a9440a4040 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- 172.21.15.38:0/3961850518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 msgr2=0x55a9440a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 0x55a9440a4460 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6568009ca0 tx=0x7f6568032f70 comp rx=0 tx=0).stop 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- 172.21.15.38:0/3961850518 shutdown_connections 2024-04-22T01:15:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4a30 0x55a9440a9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- 172.21.15.38:0/3961850518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4040 0x55a9440a4460 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- 172.21.15.38:0/3961850518 >> 172.21.15.38:0/3961850518 conn(0x55a94409f5b0 msgr2=0x55a94409f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- 172.21.15.38:0/3961850518 shutdown_connections 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- 172.21.15.38:0/3961850518 wait complete. 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- start start 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 Processor -- start 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a9440acaf0 con 0x55a9440a4a30 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a943fc4f10 con 0x55a9440a9ba0 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a9440acc70 con 0x55a9440a4040 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f6580232740 10 client.0 _ll_register_callbacks cb 0x55a943d1cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f65767fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f65767fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32936/0 (socket says 172.21.15.38:32936) 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.983+0000 7f65767fc640 1 -- 172.21.15.38:0/2356018734 learned_addr learned my addr 172.21.15.38:0/2356018734 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f6576ffd640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65777fe640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65767fc640 1 -- 172.21.15.38:0/2356018734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 msgr2=0x55a9440ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65767fc640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65767fc640 1 -- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 msgr2=0x55a9440b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65767fc640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65767fc640 1 -- 172.21.15.38:0/2356018734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a943f96c80 con 0x55a9440a4a30 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f65767fc640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440ae6d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f656c010a50 tx=0x7f656c010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6898057d86c2824f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f655aff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f655aff5640 1 -- 172.21.15.38:0/2356018734 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c00fab0 con 0x55a9440a4a30 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f655aff5640 1 -- 172.21.15.38:0/2356018734 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f656c00fc50 con 0x55a9440a4a30 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6568004b40 con 0x55a9440a4a30 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f655aff5640 1 -- 172.21.15.38:0/2356018734 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f656c024050 con 0x55a9440a4a30 2024-04-22T01:15:53.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f6580232740 10 client.16554 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a943d1d3f0 con 0x55a9440a4a30 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.984+0000 7f655aff5640 1 -- 172.21.15.38:0/2356018734 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6568004b40 con 0x55a9440a4a30 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f655aff5640 1 -- 172.21.15.38:0/2356018734 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f656c004c90 con 0x55a9440a4a30 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6580232740 10 client.16554 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6580232740 10 client.16554 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6580232740 20 client.16554 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6580232740 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a9440ba160 0x55a9440bc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6580232740 4 client.16554 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a944016a70 con 0x55a9440ba160 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6576ffd640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a9440ba160 0x55a9440bc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f655aff5640 10 client.16554 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.985+0000 7f6576ffd640 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a9440ba160 0x55a9440bc5a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f33858ac23cd7f7 server_cookie=58aa75addb7f6fd8 in_seq=0 out_seq=0 2024-04-22T01:15:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f655aff5640 1 -- 172.21.15.38:0/2356018734 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a944016a70 con 0x55a9440ba160 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f655aff5640 10 client.16554 handle_command_reply: tid=0 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 client.16554 shutdown 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 msgr2=0x55a9440ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440ae6d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f656c010a50 tx=0x7f656c010f10 comp rx=0 tx=0).stop 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 shutdown_connections 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a9440ba160 0x55a9440bc5a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440ae6d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f655aff5640 0 client.16554 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 --2- 172.21.15.38:0/2356018734 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 >> 172.21.15.38:0/2356018734 conn(0x55a94409f5b0 msgr2=0x55a94409e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 shutdown_connections 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.986+0000 7f6580232740 1 -- 172.21.15.38:0/2356018734 wait complete. 2024-04-22T01:15:53.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:53.987+0000 7f6580232740 20 client.16554 trim_cache size 0 max 16384 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 Processor -- start 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 -- start start 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4040 0x55a9440a9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f65781aa520 con 0x55a9440a4a30 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65781ac060 con 0x55a9440a4040 2024-04-22T01:15:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f6580232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65781aabf0 con 0x55a9440a9ba0 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32938/0 (socket says 172.21.15.38:32938) 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 -- 172.21.15.38:0/49670813 learned_addr learned my addr 172.21.15.38:0/49670813 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65777fe640 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 -- 172.21.15.38:0/49670813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 msgr2=0x55a9440a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 -- 172.21.15.38:0/49670813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4040 msgr2=0x55a9440a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4040 0x55a9440a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 -- 172.21.15.38:0/49670813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a943de77d0 con 0x55a9440a4a30 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65777fe640 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f65767fc640 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b56d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f656c019680 tx=0x7f656c019890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635a2717da120172 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f655cff9640 1 -- 172.21.15.38:0/49670813 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c00fab0 con 0x55a9440a4a30 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f655cff9640 1 -- 172.21.15.38:0/49670813 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656c02d070 con 0x55a9440a4a30 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.000+0000 7f655cff9640 1 -- 172.21.15.38:0/49670813 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c005420 con 0x55a9440a4a30 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 -- 172.21.15.38:0/49670813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 msgr2=0x55a9440b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b56d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f656c019680 tx=0x7f656c019890 comp rx=0 tx=0).stop 2024-04-22T01:15:54.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 -- 172.21.15.38:0/49670813 shutdown_connections 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a9ba0 0x55a9440a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b56d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 --2- 172.21.15.38:0/49670813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a4040 0x55a9440a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 -- 172.21.15.38:0/49670813 >> 172.21.15.38:0/49670813 conn(0x55a94409f5b0 msgr2=0x55a9440a2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 -- 172.21.15.38:0/49670813 shutdown_connections 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 -- 172.21.15.38:0/49670813 wait complete. 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 -- start start 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.001+0000 7f6580232740 1 Processor -- start 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f65781ab990 con 0x55a9440a4a30 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f65781ab2c0 con 0x55a9440a9ba0 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65781a84e0 con 0x55a9440a4040 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 10 client.0 _ll_register_callbacks cb 0x7f656c010660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:54.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32950/0 (socket says 172.21.15.38:32950) 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 -- 172.21.15.38:0/4090846664 learned_addr learned my addr 172.21.15.38:0/4090846664 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65777fe640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6576ffd640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 -- 172.21.15.38:0/4090846664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 msgr2=0x55a9440b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 -- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 msgr2=0x55a9440b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 -- 172.21.15.38:0/4090846664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9440b4360 con 0x55a9440a4a30 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65777fe640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6576ffd640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f65767fc640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b2d60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f656c000c00 tx=0x7f656c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3f49fed92c40a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f655aff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f655aff5640 1 -- 172.21.15.38:0/4090846664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c019c10 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f655aff5640 1 -- 172.21.15.38:0/4090846664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f656c019db0 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f656c02d070 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f655aff5640 1 -- 172.21.15.38:0/4090846664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f656c005120 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 10 client.16560 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.002+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9440b55d0 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f655aff5640 1 -- 172.21.15.38:0/4090846664 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656c02d070 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f655aff5640 1 -- 172.21.15.38:0/4090846664 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f656c028430 con 0x55a9440a4a30 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6580232740 10 client.16560 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6580232740 10 client.16560 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6580232740 20 client.16560 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6580232740 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a944120ae0 0x55a944122f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6580232740 4 client.16560 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 --> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a944010dc0 con 0x55a944120ae0 2024-04-22T01:15:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.003+0000 7f6576ffd640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a944120ae0 0x55a944122f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f655aff5640 10 client.16560 ms_handle_connect on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6576ffd640 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a944120ae0 0x55a944122f00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=651dbe8ef31df24d server_cookie=135f15e341b013c9 in_seq=0 out_seq=0 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f655aff5640 1 -- 172.21.15.38:0/4090846664 <== mds.1 v2:172.21.15.111:6836/1636882248 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a944010dc0 con 0x55a944120ae0 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f655aff5640 10 client.16560 handle_command_reply: tid=0 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 client.16560 shutdown 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 msgr2=0x55a9440b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b2d60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f656c000c00 tx=0x7f656c00fce0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 shutdown_connections 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:6836/1636882248,v1:172.21.15.111:6837/1636882248] conn(0x55a944120ae0 0x55a944122f00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9440a9ba0 0x55a9440b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9440a4a30 0x55a9440b2d60 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 --2- 172.21.15.38:0/4090846664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a9440a4040 0x55a9440b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f655aff5640 0 client.16560 ms_handle_reset on v2:172.21.15.111:6836/1636882248 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 >> 172.21.15.38:0/4090846664 conn(0x55a94409f5b0 msgr2=0x55a9440a2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 shutdown_connections 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 1 -- 172.21.15.38:0/4090846664 wait complete. 2024-04-22T01:15:54.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.004+0000 7f6580232740 20 client.16560 trim_cache size 0 max 16384 2024-04-22T01:15:54.009 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:54.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f65600a2b90 msgr2=0x7f65600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f65600a2b90 0x7f65600a5050 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6568036000 tx=0x7f6568042000 comp rx=0 tx=0).stop 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 msgr2=0x7f6578197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 0x7f6578197a80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f657000e9f0 tx=0x7f657000edf0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 shutdown_connections 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f65600a2b90 0x7f65600a5050 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65781074e0 0x7f657819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6578104bb0 0x7f6578197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 --2- 172.21.15.38:0/1611177601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6578102280 0x7f6578197a80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 >> 172.21.15.38:0/1611177601 conn(0x7f65780fbc30 msgr2=0x7f6578102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 shutdown_connections 2024-04-22T01:15:54.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.005+0000 7f657d56c640 1 -- 172.21.15.38:0/1611177601 wait complete. 2024-04-22T01:15:54.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:53 smithi038 ceph-mon[28043]: pgmap v461: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 107 op/s 2024-04-22T01:15:54.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:53 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3836826798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:15:54.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 -- 172.21.15.38:0/4065757208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a41073e0 msgr2=0x7f55a41098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 --2- 172.21.15.38:0/4065757208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a41073e0 0x7f55a41098a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f559c009ca0 tx=0x7f559c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 -- 172.21.15.38:0/4065757208 shutdown_connections 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 --2- 172.21.15.38:0/4065757208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a41073e0 0x7f55a41098a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 --2- 172.21.15.38:0/4065757208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55a40ffe50 0x7f55a4106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 --2- 172.21.15.38:0/4065757208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a40ff4f0 0x7f55a40ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 -- 172.21.15.38:0/4065757208 >> 172.21.15.38:0/4065757208 conn(0x7f55a40faf40 msgr2=0x7f55a40fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 -- 172.21.15.38:0/4065757208 shutdown_connections 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.120+0000 7f55a9572640 1 -- 172.21.15.38:0/4065757208 wait complete. 2024-04-22T01:15:54.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 Processor -- start 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 -- start start 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55a40ff4f0 0x7f55a41979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 0x7f55a4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a41073e0 0x7f55a419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55a410d3d0 con 0x7f55a40ffe50 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f55a410d250 con 0x7f55a40ff4f0 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a9572640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55a410d550 con 0x7f55a41073e0 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 0x7f55a4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 0x7f55a4197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32956/0 (socket says 172.21.15.38:32956) 2024-04-22T01:15:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 -- 172.21.15.38:0/3878269968 learned_addr learned my addr 172.21.15.38:0/3878269968 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f55a37fe640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a41073e0 0x7f55a419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 -- 172.21.15.38:0/3878269968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a41073e0 msgr2=0x7f55a419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a41073e0 0x7f55a419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 -- 172.21.15.38:0/3878269968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55a40ff4f0 msgr2=0x7f55a41979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55a40ff4f0 0x7f55a41979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.121+0000 7f559bfff640 1 -- 172.21.15.38:0/3878269968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a419f5d0 con 0x7f55a40ffe50 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.122+0000 7f559bfff640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 0x7f55a4197ef0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f558c00d770 tx=0x7f558c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.122+0000 7f55a0ff9640 1 -- 172.21.15.38:0/3878269968 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c017070 con 0x7f55a40ffe50 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.122+0000 7f55a0ff9640 1 -- 172.21.15.38:0/3878269968 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f558c009c30 con 0x7f55a40ffe50 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.122+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a419f8c0 con 0x7f55a40ffe50 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.122+0000 7f55a0ff9640 1 -- 172.21.15.38:0/3878269968 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c003dd0 con 0x7f55a40ffe50 2024-04-22T01:15:54.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.122+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a419fe60 con 0x7f55a40ffe50 2024-04-22T01:15:54.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.124+0000 7f55a0ff9640 1 -- 172.21.15.38:0/3878269968 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f558c003fb0 con 0x7f55a40ffe50 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.124+0000 7f55ac230740 1 Processor -- start 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.124+0000 7f55ac230740 1 -- start start 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55ac230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f7318b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55ac230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318ba30 0x560f73190530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55ac230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f73193030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55ac230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f730f0530 con 0x560f7318b040 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55ac230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560f73098830 con 0x560f7318ba30 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55ac230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560f730ec4f0 con 0x560f73190ba0 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f7318b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f7318b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32958/0 (socket says 172.21.15.38:32958) 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 -- 172.21.15.38:0/2373581016 learned_addr learned my addr 172.21.15.38:0/2373581016 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a0ff9640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f55900a2c10 0x7f55900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318ba30 0x560f73190530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f55900a2c10 0x7f55900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a0ff9640 1 -- 172.21.15.38:0/3878269968 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f558c0cee30 con 0x7f55a40ffe50 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f559bfff640 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f73193030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 -- 172.21.15.38:0/2373581016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 msgr2=0x560f73193030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f73193030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 -- 172.21.15.38:0/2373581016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318ba30 msgr2=0x560f73190530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318ba30 0x560f73190530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a37fe640 1 -- 172.21.15.38:0/2373581016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f730bc2c0 con 0x560f7318b040 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.125+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318ba30 0x560f73190530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55a37fe640 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f7318b460 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f559c0324c0 tx=0x7f559c032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d14a8013a2d53b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55997fa640 1 -- 172.21.15.38:0/2373581016 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c039040 con 0x560f7318b040 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55997fa640 1 -- 172.21.15.38:0/2373581016 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c032870 con 0x560f7318b040 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55997fa640 1 -- 172.21.15.38:0/2373581016 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c034710 con 0x560f7318b040 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55ac230740 1 -- 172.21.15.38:0/2373581016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 msgr2=0x560f7318b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55ac230740 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f7318b460 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f559c0324c0 tx=0x7f559c032e20 comp rx=0 tx=0).stop 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.126+0000 7f55a37fe640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f55900a2c10 0x7f55900a50d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f559c04c000 tx=0x7f559c048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- 172.21.15.38:0/2373581016 shutdown_connections 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f73193030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318ba30 0x560f73190530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 --2- 172.21.15.38:0/2373581016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f7318b460 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- 172.21.15.38:0/2373581016 >> 172.21.15.38:0/2373581016 conn(0x560f731865b0 msgr2=0x560f731869b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- 172.21.15.38:0/2373581016 shutdown_connections 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- 172.21.15.38:0/2373581016 wait complete. 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- start start 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 Processor -- start 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73195260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f731957a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 0x560f7319c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f73193b50 con 0x560f73190ba0 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560f730abf10 con 0x560f7318b040 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560f73193cd0 con 0x560f7318ba30 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55ac230740 10 client.0 _ll_register_callbacks cb 0x560f72e03be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:54.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55a37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 0x560f7319c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55a37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 0x560f7319c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32966/0 (socket says 172.21.15.38:32966) 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f55a37fe640 1 -- 172.21.15.38:0/2183162984 learned_addr learned my addr 172.21.15.38:0/2183162984 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.127+0000 7f559bfff640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f731957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a37fe640 1 -- 172.21.15.38:0/2183162984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 msgr2=0x560f731957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a37fe640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f731957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a37fe640 1 -- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 msgr2=0x560f73195260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a37fe640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73195260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a37fe640 1 -- 172.21.15.38:0/2183162984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f7307dc80 con 0x560f73190ba0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55a37fe640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 0x560f7319c7e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f559c02c700 tx=0x7f559c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed370778babe97ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f558a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f558a7fc640 1 -- 172.21.15.38:0/2183162984 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c058070 con 0x560f73190ba0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f558a7fc640 1 -- 172.21.15.38:0/2183162984 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f559c0344b0 con 0x560f73190ba0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f559c032870 con 0x560f73190ba0 2024-04-22T01:15:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f558a7fc640 1 -- 172.21.15.38:0/2183162984 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f559c064050 con 0x560f73190ba0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55ac230740 10 client.16572 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f72e043f0 con 0x560f73190ba0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.128+0000 7f558a7fc640 1 -- 172.21.15.38:0/2183162984 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c032870 con 0x560f73190ba0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f558a7fc640 1 -- 172.21.15.38:0/2183162984 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f559c04c430 con 0x560f73190ba0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55ac230740 10 client.16572 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55ac230740 10 client.16572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55ac230740 20 client.16572 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55ac230740 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f731a0cd0 0x560f731a3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55ac230740 4 client.16572 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f730fda70 con 0x560f731a0cd0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f731a0cd0 0x560f731a3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f731a0cd0 0x560f731a3100 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66499ab93b9e5f1 server_cookie=d1b690bf8adb3d75 in_seq=0 out_seq=0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.129+0000 7f558a7fc640 10 client.16572 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f558a7fc640 1 -- 172.21.15.38:0/2183162984 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f730fda70 con 0x560f731a0cd0 2024-04-22T01:15:54.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f558a7fc640 10 client.16572 handle_command_reply: tid=0 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 client.16572 shutdown 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 msgr2=0x560f7319c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 0x560f7319c7e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f559c02c700 tx=0x7f559c009540 comp rx=0 tx=0).stop 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 shutdown_connections 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f731a0cd0 0x560f731a3100 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f73190ba0 0x560f7319c7e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f731957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f558a7fc640 0 client.16572 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 --2- 172.21.15.38:0/2183162984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 >> 172.21.15.38:0/2183162984 conn(0x560f731865b0 msgr2=0x560f73185850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 shutdown_connections 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 1 -- 172.21.15.38:0/2183162984 wait complete. 2024-04-22T01:15:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.130+0000 7f55ac230740 20 client.16572 trim_cache size 0 max 16384 2024-04-22T01:15:54.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:53 smithi111 ceph-mon[31057]: pgmap v461: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 107 op/s 2024-04-22T01:15:54.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:53 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3836826798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:15:54.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.143+0000 7f55ac230740 1 Processor -- start 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 -- start start 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f731856d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f7319c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f73190ba0 0x560f7318c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41aa420 con 0x560f7318b040 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41abf60 con 0x560f73190ba0 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55ac230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41aaaf0 con 0x560f7318ba30 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f731856d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f731856d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32984/0 (socket says 172.21.15.38:32984) 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 -- 172.21.15.38:0/1423296778 learned_addr learned my addr 172.21.15.38:0/1423296778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f559bfff640 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f7319c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a37fe640 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f73190ba0 0x560f7318c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 -- 172.21.15.38:0/1423296778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 msgr2=0x560f7319c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f7319c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 -- 172.21.15.38:0/1423296778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f73190ba0 msgr2=0x560f7318c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f73190ba0 0x560f7318c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 -- 172.21.15.38:0/1423296778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f72ece7d0 con 0x560f7318b040 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a37fe640 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f73190ba0 0x560f7318c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.144+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f731856d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x560f731a3920 tx=0x7f5594005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31db5279a49d8c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f5598ff9640 1 -- 172.21.15.38:0/1423296778 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55940120d0 con 0x560f7318b040 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f5598ff9640 1 -- 172.21.15.38:0/1423296778 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5594012270 con 0x560f7318b040 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f5598ff9640 1 -- 172.21.15.38:0/1423296778 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5594010a70 con 0x560f7318b040 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 -- 172.21.15.38:0/1423296778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 msgr2=0x560f731856d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f731856d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x560f731a3920 tx=0x7f5594005ce0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 -- 172.21.15.38:0/1423296778 shutdown_connections 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f73190ba0 0x560f7318c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f7318ba30 0x560f7319c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 --2- 172.21.15.38:0/1423296778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318b040 0x560f731856d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 -- 172.21.15.38:0/1423296778 >> 172.21.15.38:0/1423296778 conn(0x560f731865b0 msgr2=0x560f73184fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 -- 172.21.15.38:0/1423296778 shutdown_connections 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.145+0000 7f55ac230740 1 -- 172.21.15.38:0/1423296778 wait complete. 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 -- start start 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 Processor -- start 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73199b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 0x560f7319a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f7319b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41ab890 con 0x560f7318ba30 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41ab1c0 con 0x560f7318b040 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41a83e0 con 0x560f73190ba0 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55ac230740 10 client.0 _ll_register_callbacks cb 0x7f5594005f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 0x560f7319a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 0x560f7319a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32992/0 (socket says 172.21.15.38:32992) 2024-04-22T01:15:54.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 -- 172.21.15.38:0/4169056407 learned_addr learned my addr 172.21.15.38:0/4169056407 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55a37fe640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f7319b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 -- 172.21.15.38:0/4169056407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 msgr2=0x560f7319b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f7319b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 -- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 msgr2=0x560f73199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 -- 172.21.15.38:0/4169056407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f7319b6b0 con 0x560f7318ba30 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f55a37fe640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f7319b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f559bfff640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 0x560f7319a0b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f558c09afc0 tx=0x7f558c009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e120cd82dcb430d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.146+0000 7f558a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f558a7fc640 1 -- 172.21.15.38:0/4169056407 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c020040 con 0x560f7318ba30 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f558a7fc640 1 -- 172.21.15.38:0/4169056407 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f558c097930 con 0x560f7318ba30 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5594012270 con 0x560f7318ba30 2024-04-22T01:15:54.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f558a7fc640 1 -- 172.21.15.38:0/4169056407 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f558c09f670 con 0x560f7318ba30 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 10 client.16578 fetch_fsmap learned FSMap version 28 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f7319c990 con 0x560f7318ba30 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f558a7fc640 1 -- 172.21.15.38:0/4169056407 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5594012270 con 0x560f7318ba30 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f558a7fc640 1 -- 172.21.15.38:0/4169056407 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f558c09f160 con 0x560f7318ba30 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 10 client.16578 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 10 client.16578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 20 client.16578 populate_metadata read hostname 'smithi038' 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f732078b0 0x560f73209cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 4 client.16578 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.147+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 --> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f730f7dc0 con 0x560f732078b0 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f732078b0 0x560f73209cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f558a7fc640 10 client.16578 ms_handle_connect on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55a2ffd640 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f732078b0 0x560f73209cd0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b184193be941b6a server_cookie=f7f217518ded4286 in_seq=0 out_seq=0 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f558a7fc640 1 -- 172.21.15.38:0/4169056407 <== mds.0 v2:172.21.15.111:6838/3001874105 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f730f7dc0 con 0x560f732078b0 2024-04-22T01:15:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f558a7fc640 10 client.16578 handle_command_reply: tid=0 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 client.16578 shutdown 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 msgr2=0x560f7319a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 0x560f7319a0b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f558c09afc0 tx=0x7f558c009de0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 shutdown_connections 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:6838/3001874105,v1:172.21.15.111:6839/3001874105] conn(0x560f732078b0 0x560f73209cd0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f73190ba0 0x560f7319b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f558a7fc640 0 client.16578 ms_handle_reset on v2:172.21.15.111:6838/3001874105 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f7318ba30 0x560f7319a0b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 --2- 172.21.15.38:0/4169056407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f7318b040 0x560f73199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 >> 172.21.15.38:0/4169056407 conn(0x560f731865b0 msgr2=0x560f7318a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 shutdown_connections 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 1 -- 172.21.15.38:0/4169056407 wait complete. 2024-04-22T01:15:54.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.148+0000 7f55ac230740 20 client.16578 trim_cache size 0 max 16384 2024-04-22T01:15:54.155 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T01:15:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f55900a2c10 msgr2=0x7f55900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f55900a2c10 0x7f55900a50d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f559c04c000 tx=0x7f559c048000 comp rx=0 tx=0).stop 2024-04-22T01:15:54.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 msgr2=0x7f55a4197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 0x7f55a4197ef0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f558c00d770 tx=0x7f558c00dc30 comp rx=0 tx=0).stop 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 shutdown_connections 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f55900a2c10 0x7f55900a50d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a41073e0 0x7f55a419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55a40ffe50 0x7f55a4197ef0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 --2- 172.21.15.38:0/3878269968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55a40ff4f0 0x7f55a41979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 >> 172.21.15.38:0/3878269968 conn(0x7f55a40faf40 msgr2=0x7f55a4102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 shutdown_connections 2024-04-22T01:15:54.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:54.149+0000 7f55a9572640 1 -- 172.21.15.38:0/3878269968 wait complete. 2024-04-22T01:15:54.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:53 smithi203 ceph-mon[31825]: pgmap v461: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 107 op/s 2024-04-22T01:15:54.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:53 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3836826798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:15:55.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:54 smithi111 ceph-mon[31057]: pgmap v462: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 33 op/s 2024-04-22T01:15:55.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:54 smithi038 ceph-mon[28043]: pgmap v462: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 33 op/s 2024-04-22T01:15:55.160 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-22T01:15:55.160 INFO:tasks.fwd_scrub:done joining 2024-04-22T01:15:55.160 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-22T01:15:55.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:15:55.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:54 smithi203 ceph-mon[31825]: pgmap v462: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 33 op/s 2024-04-22T01:15:55.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 -- 172.21.15.38:0/418250184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c01022c0 msgr2=0x7fd3c01046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 --2- 172.21.15.38:0/418250184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c01022c0 0x7fd3c01046b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009900 tx=0x7fd3b002bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:55.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 -- 172.21.15.38:0/418250184 shutdown_connections 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 --2- 172.21.15.38:0/418250184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3c0107520 0x7fd3c0109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 --2- 172.21.15.38:0/418250184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 --2- 172.21.15.38:0/418250184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c01022c0 0x7fd3c01046b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 -- 172.21.15.38:0/418250184 >> 172.21.15.38:0/418250184 conn(0x7fd3c00fbc30 msgr2=0x7fd3c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 -- 172.21.15.38:0/418250184 shutdown_connections 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 -- 172.21.15.38:0/418250184 wait complete. 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.291+0000 7fd3c560e640 1 Processor -- start 2024-04-22T01:15:55.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 -- start start 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3c01022c0 0x7fd3c0101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3c0107520 0x7fd3c01007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c010d410 con 0x7fd3c0104bf0 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c010d290 con 0x7fd3c0107520 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3c560e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c010d590 con 0x7fd3c01022c0 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0100270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33006/0 (socket says 172.21.15.38:33006) 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 -- 172.21.15.38:0/3281873174 learned_addr learned my addr 172.21.15.38:0/3281873174 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3beffd640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3c01022c0 0x7fd3c0101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3bf7fe640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3c0107520 0x7fd3c01007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 -- 172.21.15.38:0/3281873174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3c01022c0 msgr2=0x7fd3c0101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3c01022c0 0x7fd3c0101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 -- 172.21.15.38:0/3281873174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3c0107520 msgr2=0x7fd3c01007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3c0107520 0x7fd3c01007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 -- 172.21.15.38:0/3281873174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c0101040 con 0x7fd3c0104bf0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.292+0000 7fd3be7fc640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0100270 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400c970 tx=0x7fd3b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.293+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4014070 con 0x7fd3c0104bf0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.293+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b4004440 con 0x7fd3c0104bf0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.293+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c01a1b00 con 0x7fd3c0104bf0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.293+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400f450 con 0x7fd3c0104bf0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.293+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c01a2050 con 0x7fd3c0104bf0 2024-04-22T01:15:55.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.294+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c00fd720 con 0x7fd3c0104bf0 2024-04-22T01:15:55.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.295+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd3b4017070 con 0x7fd3c0104bf0 2024-04-22T01:15:55.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.295+0000 7fd3a7fff640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3a80a2b90 0x7fd3a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.296+0000 7fd3beffd640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3a80a2b90 0x7fd3a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.296+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd3b40cf300 con 0x7fd3c0104bf0 2024-04-22T01:15:55.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.296+0000 7fd3beffd640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3a80a2b90 0x7fd3a80a5050 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005ab0 tx=0x7fd3b0004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.300+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd3b401e020 con 0x7fd3c0104bf0 2024-04-22T01:15:55.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.491+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3c0062340 con 0x7fd3c0104bf0 2024-04-22T01:15:55.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.493+0000 7fd3a7fff640 1 -- 172.21.15.38:0/3281873174 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7fd3b409be50 con 0x7fd3c0104bf0 2024-04-22T01:15:55.492 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:15:55.492 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3a80a2b90 msgr2=0x7fd3a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3a80a2b90 0x7fd3a80a5050 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005ab0 tx=0x7fd3b0004f10 comp rx=0 tx=0).stop 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 msgr2=0x7fd3c0100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0100270 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400c970 tx=0x7fd3b400ce30 comp rx=0 tx=0).stop 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 shutdown_connections 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3a80a2b90 0x7fd3a80a5050 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3c0107520 0x7fd3c01007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3c0104bf0 0x7fd3c0100270 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 --2- 172.21.15.38:0/3281873174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3c01022c0 0x7fd3c0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 >> 172.21.15.38:0/3281873174 conn(0x7fd3c00fbc30 msgr2=0x7fd3c0102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 shutdown_connections 2024-04-22T01:15:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.495+0000 7fd3c560e640 1 -- 172.21.15.38:0/3281873174 wait complete. 2024-04-22T01:15:55.506 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:15:55.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:15:55.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.612+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3190744977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 msgr2=0x7fd1b8102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.612+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3190744977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8102e10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009900 tx=0x7fd1ac02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3190744977 shutdown_connections 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3190744977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b8103350 0x7fd1b8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3190744977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8102e10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3190744977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1b81017a0 0x7fd1b8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3190744977 >> 172.21.15.38:0/3190744977 conn(0x7fd1b80fcd10 msgr2=0x7fd1b80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3190744977 shutdown_connections 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3190744977 wait complete. 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 Processor -- start 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.613+0000 7fd1bd1d7640 1 -- start start 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1bd1d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b81017a0 0x7fd1b8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1bd1d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1bd1d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1b8103350 0x7fd1b819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1bd1d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b810d400 con 0x7fd1b8102990 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1bd1d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b810d280 con 0x7fd1b8103350 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1bd1d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b810d580 con 0x7fd1b81017a0 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33026/0 (socket says 172.21.15.38:33026) 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 -- 172.21.15.38:0/3051375674 learned_addr learned my addr 172.21.15.38:0/3051375674 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b6d9d640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b81017a0 0x7fd1b8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b759e640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1b8103350 0x7fd1b819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 -- 172.21.15.38:0/3051375674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b81017a0 msgr2=0x7fd1b8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b81017a0 0x7fd1b8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 -- 172.21.15.38:0/3051375674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1b8103350 msgr2=0x7fd1b819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1b8103350 0x7fd1b819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 -- 172.21.15.38:0/3051375674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b819f6c0 con 0x7fd1b8102990 2024-04-22T01:15:55.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd1b659c640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8197fe0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009230 tx=0x7fd1ac005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.614+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac041070 con 0x7fd1b8102990 2024-04-22T01:15:55.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.615+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b819f950 con 0x7fd1b8102990 2024-04-22T01:15:55.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.615+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1ac0054e0 con 0x7fd1b8102990 2024-04-22T01:15:55.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.615+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac034440 con 0x7fd1b8102990 2024-04-22T01:15:55.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.615+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b819fe90 con 0x7fd1b8102990 2024-04-22T01:15:55.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.616+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b8108b10 con 0x7fd1b8102990 2024-04-22T01:15:55.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.617+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd1ac044070 con 0x7fd1b8102990 2024-04-22T01:15:55.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.617+0000 7fd19ffff640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd1a00a2b90 0x7fd1a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.618+0000 7fd1b6d9d640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd1a00a2b90 0x7fd1a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.618+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd1ac0f1030 con 0x7fd1b8102990 2024-04-22T01:15:55.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.618+0000 7fd1b6d9d640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd1a00a2b90 0x7fd1a00a5050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd1b81027f0 tx=0x7fd1a800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.622+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd1ac0bd570 con 0x7fd1b8102990 2024-04-22T01:15:55.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.806+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1b8062340 con 0x7fd1b8102990 2024-04-22T01:15:55.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.807+0000 7fd19ffff640 1 -- 172.21.15.38:0/3051375674 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd1ac0c1870 con 0x7fd1b8102990 2024-04-22T01:15:55.806 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.808+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd1a00a2b90 msgr2=0x7fd1a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.808+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd1a00a2b90 0x7fd1a00a5050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd1b81027f0 tx=0x7fd1a800b390 comp rx=0 tx=0).stop 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.808+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 msgr2=0x7fd1b8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.808+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8197fe0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009230 tx=0x7fd1ac005dd0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 shutdown_connections 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd1a00a2b90 0x7fd1a00a5050 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1b8103350 0x7fd1b819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1b8102990 0x7fd1b8197fe0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 --2- 172.21.15.38:0/3051375674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b81017a0 0x7fd1b8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 >> 172.21.15.38:0/3051375674 conn(0x7fd1b80fcd10 msgr2=0x7fd1b8105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 shutdown_connections 2024-04-22T01:15:55.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.809+0000 7fd1bd1d7640 1 -- 172.21.15.38:0/3051375674 wait complete. 2024-04-22T01:15:55.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:15:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.924+0000 7f21a45f7640 1 -- 172.21.15.38:0/3230426690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c102200 msgr2=0x7f219c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.924+0000 7f21a45f7640 1 --2- 172.21.15.38:0/3230426690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c102200 0x7f219c1045f0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2198009900 tx=0x7f219802bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- 172.21.15.38:0/3230426690 shutdown_connections 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 --2- 172.21.15.38:0/3230426690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c107460 0x7f219c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 --2- 172.21.15.38:0/3230426690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219c104b30 0x7f219c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 --2- 172.21.15.38:0/3230426690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c102200 0x7f219c1045f0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- 172.21.15.38:0/3230426690 >> 172.21.15.38:0/3230426690 conn(0x7f219c0fbbf0 msgr2=0x7f219c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- 172.21.15.38:0/3230426690 shutdown_connections 2024-04-22T01:15:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- 172.21.15.38:0/3230426690 wait complete. 2024-04-22T01:15:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 Processor -- start 2024-04-22T01:15:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- start start 2024-04-22T01:15:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c102200 0x7f219c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 0x7f219c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219c107460 0x7f219c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f219c10d430 con 0x7f219c104b30 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f219c10d2b0 con 0x7f219c102200 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a45f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f219c10d5b0 con 0x7f219c107460 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a1b92640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 0x7f219c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a1b92640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 0x7f219c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33048/0 (socket says 172.21.15.38:33048) 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.925+0000 7f21a1b92640 1 -- 172.21.15.38:0/743910278 learned_addr learned my addr 172.21.15.38:0/743910278 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a2b94640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219c107460 0x7f219c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a2393640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c102200 0x7f219c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a1b92640 1 -- 172.21.15.38:0/743910278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219c107460 msgr2=0x7f219c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a1b92640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219c107460 0x7f219c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a1b92640 1 -- 172.21.15.38:0/743910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c102200 msgr2=0x7f219c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a1b92640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c102200 0x7f219c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a1b92640 1 -- 172.21.15.38:0/743910278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219c19f790 con 0x7f219c104b30 2024-04-22T01:15:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a1b92640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 0x7f219c1980b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f218c00c970 tx=0x7f218c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218c007bb0 con 0x7f219c104b30 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f218c007d50 con 0x7f219c104b30 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f219c19fa80 con 0x7f219c104b30 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218c0187a0 con 0x7f219c104b30 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.926+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219c19ffd0 con 0x7f219c104b30 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.927+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f219c0fd6e0 con 0x7f219c104b30 2024-04-22T01:15:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.928+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f218c00f450 con 0x7f219c104b30 2024-04-22T01:15:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.929+0000 7f21937fe640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21880a2c10 0x7f21880a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.929+0000 7f21a2393640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21880a2c10 0x7f21880a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.929+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f218c0cf850 con 0x7f219c104b30 2024-04-22T01:15:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.929+0000 7f21a2393640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21880a2c10 0x7f21880a50d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2198006010 tx=0x7f2198000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:55.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:55.932+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f218c014030 con 0x7f219c104b30 2024-04-22T01:15:56.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.115+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f219c02d3c0 con 0x7f219c104b30 2024-04-22T01:15:56.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.116+0000 7f21937fe640 1 -- 172.21.15.38:0/743910278 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f218c09bdd0 con 0x7f219c104b30 2024-04-22T01:15:56.115 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21880a2c10 msgr2=0x7f21880a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21880a2c10 0x7f21880a50d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2198006010 tx=0x7f2198000bc0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 msgr2=0x7f219c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 0x7f219c1980b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f218c00c970 tx=0x7f218c00ce30 comp rx=0 tx=0).stop 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 shutdown_connections 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f21880a2c10 0x7f21880a50d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f219c107460 0x7f219c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f219c104b30 0x7f219c1980b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 --2- 172.21.15.38:0/743910278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f219c102200 0x7f219c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 >> 172.21.15.38:0/743910278 conn(0x7f219c0fbbf0 msgr2=0x7f219c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 shutdown_connections 2024-04-22T01:15:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.118+0000 7f21a45f7640 1 -- 172.21.15.38:0/743910278 wait complete. 2024-04-22T01:15:56.126 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:55 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3281873174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:15:56.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:15:56.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:55 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3281873174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:15:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 -- 172.21.15.38:0/3820574101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0102980 msgr2=0x7f20b0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 --2- 172.21.15.38:0/3820574101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0102980 0x7f20b0102e00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009900 tx=0x7f20a002bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 -- 172.21.15.38:0/3820574101 shutdown_connections 2024-04-22T01:15:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 --2- 172.21.15.38:0/3820574101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0103340 0x7f20b0109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 --2- 172.21.15.38:0/3820574101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0102980 0x7f20b0102e00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 --2- 172.21.15.38:0/3820574101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b0101790 0x7f20b0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 -- 172.21.15.38:0/3820574101 >> 172.21.15.38:0/3820574101 conn(0x7f20b00fcd20 msgr2=0x7f20b00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 -- 172.21.15.38:0/3820574101 shutdown_connections 2024-04-22T01:15:56.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.234+0000 7f20b845f640 1 -- 172.21.15.38:0/3820574101 wait complete. 2024-04-22T01:15:56.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 Processor -- start 2024-04-22T01:15:56.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 -- start start 2024-04-22T01:15:56.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0101790 0x7f20b0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b0102980 0x7f20b0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 0x7f20b019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f20b010d440 con 0x7f20b0103340 2024-04-22T01:15:56.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f20b010d2c0 con 0x7f20b0102980 2024-04-22T01:15:56.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b845f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20b010d5c0 con 0x7f20b0101790 2024-04-22T01:15:56.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b69fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 0x7f20b019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b69fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 0x7f20b019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33058/0 (socket says 172.21.15.38:33058) 2024-04-22T01:15:56.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b69fc640 1 -- 172.21.15.38:0/2701131514 learned_addr learned my addr 172.21.15.38:0/2701131514 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:56.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b61fb640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0101790 0x7f20b0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.235+0000 7f20b59fa640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b0102980 0x7f20b0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b69fc640 1 -- 172.21.15.38:0/2701131514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0101790 msgr2=0x7f20b0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b69fc640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0101790 0x7f20b0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b69fc640 1 -- 172.21.15.38:0/2701131514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b0102980 msgr2=0x7f20b0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b69fc640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b0102980 0x7f20b0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b69fc640 1 -- 172.21.15.38:0/2701131514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b019f6d0 con 0x7f20b0103340 2024-04-22T01:15:56.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b61fb640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0101790 0x7f20b0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:15:56.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b69fc640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 0x7f20b019f060 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f20a800cce0 tx=0x7f20a800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a800f450 con 0x7f20b0103340 2024-04-22T01:15:56.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a800fad0 con 0x7f20b0103340 2024-04-22T01:15:56.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b019f9c0 con 0x7f20b0103340 2024-04-22T01:15:56.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8018c10 con 0x7f20b0103340 2024-04-22T01:15:56.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.236+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b019fee0 con 0x7f20b0103340 2024-04-22T01:15:56.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.237+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b0108ab0 con 0x7f20b0103340 2024-04-22T01:15:56.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.238+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f20a800f5f0 con 0x7f20b0103340 2024-04-22T01:15:56.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.239+0000 7f20a77fe640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f209c0a2c10 0x7f209c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.239+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f20a80cfb00 con 0x7f20b0103340 2024-04-22T01:15:56.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.239+0000 7f20b61fb640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f209c0a2c10 0x7f209c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.239+0000 7f20b61fb640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f209c0a2c10 0x7f209c0a50d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0058c0 tx=0x7f20ac005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.243+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f20a8014030 con 0x7f20b0103340 2024-04-22T01:15:56.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:55 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3281873174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:15:56.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.427+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f20b0062340 con 0x7f20b0103340 2024-04-22T01:15:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.427+0000 7f20a77fe640 1 -- 172.21.15.38:0/2701131514 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f20a809c690 con 0x7f20b0103340 2024-04-22T01:15:56.427 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f209c0a2c10 msgr2=0x7f209c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f209c0a2c10 0x7f209c0a50d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0058c0 tx=0x7f20ac005850 comp rx=0 tx=0).stop 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 msgr2=0x7f20b019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 0x7f20b019f060 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f20a800cce0 tx=0x7f20a800a620 comp rx=0 tx=0).stop 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 shutdown_connections 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f209c0a2c10 0x7f209c0a50d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20b0103340 0x7f20b019f060 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b0102980 0x7f20b0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 --2- 172.21.15.38:0/2701131514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20b0101790 0x7f20b0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 >> 172.21.15.38:0/2701131514 conn(0x7f20b00fcd20 msgr2=0x7f20b0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 shutdown_connections 2024-04-22T01:15:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.430+0000 7f20b845f640 1 -- 172.21.15.38:0/2701131514 wait complete. 2024-04-22T01:15:56.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:15:56.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 -- 172.21.15.38:0/541750855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b8101890 msgr2=0x7fe7b8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 --2- 172.21.15.38:0/541750855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b8101890 0x7fe7b8101d10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009900 tx=0x7fe7ac02bc30 comp rx=0 tx=0).stop 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 -- 172.21.15.38:0/541750855 shutdown_connections 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 --2- 172.21.15.38:0/541750855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8102250 0x7fe7b8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 --2- 172.21.15.38:0/541750855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b8101890 0x7fe7b8101d10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 --2- 172.21.15.38:0/541750855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7b81006a0 0x7fe7b8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 -- 172.21.15.38:0/541750855 >> 172.21.15.38:0/541750855 conn(0x7fe7b80fbc30 msgr2=0x7fe7b80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 -- 172.21.15.38:0/541750855 shutdown_connections 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.546+0000 7fe7bca47640 1 -- 172.21.15.38:0/541750855 wait complete. 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 Processor -- start 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 -- start start 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 0x7fe7b8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8101890 0x7fe7b8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7b8102250 0x7fe7b819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b810d480 con 0x7fe7b81006a0 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b810d300 con 0x7fe7b8101890 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7bca47640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b810d600 con 0x7fe7b8102250 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 0x7fe7b8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 0x7fe7b8197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33078/0 (socket says 172.21.15.38:33078) 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.547+0000 7fe7b659c640 1 -- 172.21.15.38:0/402496935 learned_addr learned my addr 172.21.15.38:0/402496935 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b6d9d640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7b8102250 0x7fe7b819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b5d9b640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8101890 0x7fe7b8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b659c640 1 -- 172.21.15.38:0/402496935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7b8102250 msgr2=0x7fe7b819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b659c640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7b8102250 0x7fe7b819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b659c640 1 -- 172.21.15.38:0/402496935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8101890 msgr2=0x7fe7b8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b659c640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8101890 0x7fe7b8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b659c640 1 -- 172.21.15.38:0/402496935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b819f700 con 0x7fe7b81006a0 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b5d9b640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8101890 0x7fe7b8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7b659c640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 0x7fe7b8197ae0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800c970 tx=0x7fe7a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8004220 con 0x7fe7b81006a0 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b819f9f0 con 0x7fe7b81006a0 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7a80043c0 con 0x7fe7b81006a0 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800f450 con 0x7fe7b81006a0 2024-04-22T01:15:56.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.548+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b819ff90 con 0x7fe7b81006a0 2024-04-22T01:15:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.549+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b81079c0 con 0x7fe7b81006a0 2024-04-22T01:15:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.550+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fe7a80079b0 con 0x7fe7b81006a0 2024-04-22T01:15:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.551+0000 7fe7a77fe640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe7a00a2b90 0x7fe7a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.551+0000 7fe7b5d9b640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe7a00a2b90 0x7fe7a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.551+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fe7a80cf250 con 0x7fe7b81006a0 2024-04-22T01:15:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.551+0000 7fe7b5d9b640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe7a00a2b90 0x7fe7a00a5050 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0096c0 tx=0x7fe7ac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.555+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe7a8014030 con 0x7fe7b81006a0 2024-04-22T01:15:56.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.740+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe7b8062340 con 0x7fe7b81006a0 2024-04-22T01:15:56.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.741+0000 7fe7a77fe640 1 -- 172.21.15.38:0/402496935 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fe7a809bde0 con 0x7fe7b81006a0 2024-04-22T01:15:56.740 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe7a00a2b90 msgr2=0x7fe7a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe7a00a2b90 0x7fe7a00a5050 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0096c0 tx=0x7fe7ac02df90 comp rx=0 tx=0).stop 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 msgr2=0x7fe7b8197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 0x7fe7b8197ae0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800c970 tx=0x7fe7a800ce30 comp rx=0 tx=0).stop 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 shutdown_connections 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fe7a00a2b90 0x7fe7a00a5050 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7b8102250 0x7fe7b819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7b8101890 0x7fe7b8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 --2- 172.21.15.38:0/402496935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b81006a0 0x7fe7b8197ae0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 >> 172.21.15.38:0/402496935 conn(0x7fe7b80fbc30 msgr2=0x7fe7b8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 shutdown_connections 2024-04-22T01:15:56.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.743+0000 7fe7bca47640 1 -- 172.21.15.38:0/402496935 wait complete. 2024-04-22T01:15:56.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 -- 172.21.15.38:0/667607074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 msgr2=0x7f5244108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 --2- 172.21.15.38:0/667607074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f5244108ae0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f523c009ca0 tx=0x7f523c02c0f0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 -- 172.21.15.38:0/667607074 shutdown_connections 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 --2- 172.21.15.38:0/667607074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f5244108ae0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 --2- 172.21.15.38:0/667607074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5244101890 0x7f5244101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 --2- 172.21.15.38:0/667607074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52441006a0 0x7f5244100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 -- 172.21.15.38:0/667607074 >> 172.21.15.38:0/667607074 conn(0x7f52440fbc30 msgr2=0x7f52440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:56.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 -- 172.21.15.38:0/667607074 shutdown_connections 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 -- 172.21.15.38:0/667607074 wait complete. 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.857+0000 7f52499d7640 1 Processor -- start 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 -- start start 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52441006a0 0x7f5244197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5244101890 0x7f5244197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f524419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f524410d440 con 0x7f5244102250 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f524410d2c0 con 0x7f52441006a0 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f524410d5c0 con 0x7f5244101890 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f524419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f524419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39688/0 (socket says 172.21.15.38:39688) 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 -- 172.21.15.38:0/1537214883 learned_addr learned my addr 172.21.15.38:0/1537214883 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f5242ffd640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52441006a0 0x7f5244197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52427fc640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5244101890 0x7f5244197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 -- 172.21.15.38:0/1537214883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5244101890 msgr2=0x7f5244197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5244101890 0x7f5244197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 -- 172.21.15.38:0/1537214883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52441006a0 msgr2=0x7f5244197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52441006a0 0x7f5244197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 -- 172.21.15.38:0/1537214883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524419f660 con 0x7f5244102250 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52437fe640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f524419eff0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f523c02c700 tx=0x7f523c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c039040 con 0x7f5244102250 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523c044070 con 0x7f5244102250 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.858+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524419f8f0 con 0x7f5244102250 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.859+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c0042a0 con 0x7f5244102250 2024-04-22T01:15:56.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.859+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f524419fde0 con 0x7f5244102250 2024-04-22T01:15:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.859+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52441078f0 con 0x7f5244102250 2024-04-22T01:15:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.861+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f523c003970 con 0x7f5244102250 2024-04-22T01:15:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.861+0000 7f52489d5640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f522c0a2b90 0x7f522c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:15:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.861+0000 7f5242ffd640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f522c0a2b90 0x7f522c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:15:56.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.861+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f523c0f1880 con 0x7f5244102250 2024-04-22T01:15:56.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.862+0000 7f5242ffd640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f522c0a2b90 0x7f522c0a5050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f52441016f0 tx=0x7f5234005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:15:56.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:56.866+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f523c0be4a0 con 0x7f5244102250 2024-04-22T01:15:57.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.100+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f5244062340 con 0x7f5244102250 2024-04-22T01:15:57.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.109+0000 7f52489d5640 1 -- 172.21.15.38:0/1537214883 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+740368 (secure 0 0 0) 0x7f523c0bdda0 con 0x7f5244102250 2024-04-22T01:15:57.118 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 20, 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "active_gid": 14152, 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "active_name": "x", 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "active_addrs": { 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3086698001 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3086698001 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "active_addr": "172.21.15.38:6801/3086698001", 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "active_change": "2024-04-22T01:03:15.696083+0000", 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "active_mgr_features": 4540701547738038271, 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "available": true, 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "standbys": [ 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "gid": 24109, 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "z", 2024-04-22T01:15:57.120 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_features": 4540701547738038271, 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.121 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.122 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.124 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.125 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.126 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.127 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.128 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.130 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.131 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.133 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T01:15:57.135 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T01:15:57.136 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T01:15:57.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.138 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.140 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-04-22T01:15:57.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.142 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.144 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.145 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.146 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.147 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.150 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.151 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.152 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-04-22T01:15:57.153 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-04-22T01:15:57.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.155 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.156 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.159 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.161 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.162 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.163 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T01:15:57.164 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.168 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.169 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.170 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.172 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.174 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-04-22T01:15:57.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_samba": { 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_samba", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.176 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Samba/SMB container image", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.179 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.180 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-04-22T01:15:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "grafana_dashboards_path": { 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "grafana_dashboards_path", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-04-22T01:15:57.185 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-04-22T01:15:57.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.187 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-04-22T01:15:57.189 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.190 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.191 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.195 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-04-22T01:15:57.196 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.197 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-04-22T01:15:57.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-04-22T01:15:57.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.201 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.202 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-04-22T01:15:57.203 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.206 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.209 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.210 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T01:15:57.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.212 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.213 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.214 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.216 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.217 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T01:15:57.219 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.220 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.221 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.222 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.223 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.224 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.226 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.228 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.230 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.231 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T01:15:57.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.233 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.234 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.235 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T01:15:57.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T01:15:57.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.245 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.247 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.248 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.251 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.255 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.257 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T01:15:57.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.261 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.263 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.266 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.269 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.272 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T01:15:57.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.275 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.276 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.277 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T01:15:57.278 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.279 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.280 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.281 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.282 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.283 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-04-22T01:15:57.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.287 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T01:15:57.288 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.290 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.291 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.293 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.294 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.296 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.300 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-04-22T01:15:57.301 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-04-22T01:15:57.303 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.304 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.306 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.307 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.308 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.310 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.313 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.314 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-04-22T01:15:57.315 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.316 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.318 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.319 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.321 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.324 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T01:15:57.328 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.329 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.330 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.333 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-04-22T01:15:57.334 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.336 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.337 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.338 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.339 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-04-22T01:15:57.341 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.343 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.344 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.345 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.346 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.347 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.350 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.351 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.352 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.353 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.354 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.356 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.357 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-04-22T01:15:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-04-22T01:15:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-04-22T01:15:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-04-22T01:15:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.492 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.493 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.494 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.495 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.497 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.499 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.500 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.501 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.504 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.505 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.506 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.510 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.513 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.514 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.515 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.516 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.517 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.518 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.519 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "gid": 24116, 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "y", 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_features": 4540701547738038271, 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.520 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.521 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.523 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.524 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.526 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-04-22T01:15:57.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-04-22T01:15:57.529 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.530 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.531 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.532 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.533 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.534 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T01:15:57.535 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T01:15:57.537 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T01:15:57.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.541 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.542 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.543 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.544 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T01:15:57.545 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-04-22T01:15:57.546 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.547 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.548 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T01:15:57.549 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.550 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T01:15:57.551 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T01:15:57.552 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-04-22T01:15:57.555 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.556 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-04-22T01:15:57.559 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-04-22T01:15:57.560 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-04-22T01:15:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.562 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.563 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.564 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T01:15:57.565 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.566 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.567 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.568 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.569 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.570 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-04-22T01:15:57.571 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.572 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_samba": { 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_samba", 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.574 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Samba/SMB container image", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.575 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.576 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T01:15:57.577 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.578 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.579 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.580 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.581 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "grafana_dashboards_path": { 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "grafana_dashboards_path", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T01:15:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.583 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.584 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.585 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.588 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.590 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.591 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.593 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.594 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.596 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.597 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.598 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.608 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.612 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.614 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.615 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.616 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.617 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.618 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-04-22T01:15:57.619 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.622 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.623 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.624 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.625 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.628 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T01:15:57.629 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T01:15:57.630 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T01:15:57.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T01:15:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.633 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.634 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-04-22T01:15:57.635 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.638 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T01:15:57.639 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.640 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.642 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.643 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.644 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.645 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.646 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.647 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.648 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.649 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.650 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.651 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.652 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.653 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.654 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.655 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.656 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.657 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.658 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.659 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.661 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-04-22T01:15:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-04-22T01:15:57.663 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-04-22T01:15:57.664 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T01:15:57.665 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.666 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.667 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-04-22T01:15:57.671 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-04-22T01:15:57.672 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:57.673 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:57.674 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T01:15:57.675 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-04-22T01:15:57.676 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.677 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.678 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-04-22T01:15:57.679 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.681 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.682 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.683 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.685 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.686 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T01:15:57.687 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.688 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.689 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.690 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.691 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.694 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.695 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.697 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.698 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.699 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-04-22T01:15:57.700 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-04-22T01:15:57.702 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.703 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.704 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.705 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.706 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.707 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.708 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.709 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.710 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.711 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.712 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T01:15:57.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-04-22T01:15:57.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-04-22T01:15:57.715 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-04-22T01:15:57.716 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.719 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.721 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.723 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.724 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.725 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.726 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.727 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-04-22T01:15:57.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.729 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.731 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.733 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-04-22T01:15:57.735 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T01:15:57.736 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.737 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.738 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.739 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-04-22T01:15:57.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.742 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.744 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-04-22T01:15:57.745 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.747 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.748 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.750 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.751 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-04-22T01:15:57.752 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.754 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.754 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.754 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.754 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.754 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.754 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.756 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.757 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-04-22T01:15:57.759 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.760 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.761 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.762 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.765 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.766 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.768 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.770 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.771 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.773 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.774 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.776 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.777 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.778 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.779 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.780 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.781 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.782 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.783 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.787 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.788 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.789 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T01:15:57.790 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.792 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.794 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.796 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.797 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.798 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.799 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-04-22T01:15:57.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-04-22T01:15:57.801 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.802 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.803 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.804 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.805 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-04-22T01:15:57.807 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T01:15:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.810 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.811 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-04-22T01:15:57.812 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.813 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.814 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.815 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.816 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.817 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.818 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.820 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T01:15:57.821 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-04-22T01:15:57.822 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.824 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.826 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.828 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.833 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.836 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.837 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.838 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.839 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.840 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.841 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-04-22T01:15:57.843 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.845 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.846 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.848 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-04-22T01:15:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-04-22T01:15:57.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.851 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.853 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.854 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.855 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.856 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.857 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.858 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.859 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-04-22T01:15:57.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-04-22T01:15:57.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.863 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.864 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.865 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.866 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.867 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.869 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.870 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.871 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.872 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.874 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.875 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.876 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.877 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.879 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.880 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.881 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.882 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.883 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.884 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.885 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.886 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.887 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.888 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.890 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.891 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.892 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.893 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.894 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-04-22T01:15:57.895 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.896 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.897 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.898 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.899 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.902 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.903 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.904 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.905 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.906 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.907 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.908 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.909 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.910 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.911 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.912 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.913 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.914 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.915 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.916 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-04-22T01:15:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.919 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.920 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.921 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.922 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.923 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.924 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.925 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.927 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.928 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.929 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.930 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-04-22T01:15:57.931 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.932 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "modules": [ 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "dashboard", 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "iostat", 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "nfs", 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "restful" 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.933 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.935 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.936 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.937 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.938 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.939 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.940 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-04-22T01:15:57.941 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-04-22T01:15:57.942 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-04-22T01:15:57.943 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-04-22T01:15:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.945 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-04-22T01:15:57.947 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-04-22T01:15:57.949 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-04-22T01:15:57.950 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-04-22T01:15:57.951 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-04-22T01:15:57.952 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-04-22T01:15:57.953 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-04-22T01:15:57.954 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.956 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.957 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.958 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.960 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.961 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.962 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.963 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.964 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.965 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.966 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.967 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-04-22T01:15:57.968 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.969 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.970 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.971 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.972 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.973 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.974 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.975 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.976 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T01:15:57.977 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.978 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.979 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.981 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.982 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.983 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.984 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-04-22T01:15:57.985 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-04-22T01:15:57.986 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.988 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T01:15:57.989 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-04-22T01:15:57.990 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_samba": { 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_samba", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Samba/SMB container image", 2024-04-22T01:15:57.991 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "grafana_dashboards_path": { 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "grafana_dashboards_path", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-04-22T01:15:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-04-22T01:15:58.001 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-04-22T01:15:58.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.003 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.004 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T01:15:58.005 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.007 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.008 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.009 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.010 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.012 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.013 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-04-22T01:15:58.014 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.015 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.016 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.017 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.018 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.019 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.020 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.021 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.022 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.023 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T01:15:58.024 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.026 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.027 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.029 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.030 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.031 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.032 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.033 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.034 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.035 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.036 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.037 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.038 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.039 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.040 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.043 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T01:15:58.044 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.045 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.046 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.047 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.048 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.049 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.050 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.056 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.059 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T01:15:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.061 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.062 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.063 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.065 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.066 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.067 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.068 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.069 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.070 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.071 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.072 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T01:15:58.073 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T01:15:58.074 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T01:15:58.075 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-04-22T01:15:58.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-04-22T01:15:58.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-04-22T01:15:58.078 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.079 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.080 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.081 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.082 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.083 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.084 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.085 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.086 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.087 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.088 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.089 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.090 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-04-22T01:15:58.091 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.092 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T01:15:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.094 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.099 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.100 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T01:15:58.101 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.102 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.103 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.104 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:58.105 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-04-22T01:15:58.106 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.108 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.109 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.110 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.111 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.113 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.114 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.115 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-04-22T01:15:58.116 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.117 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.118 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.119 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.120 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.123 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-04-22T01:15:58.124 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.125 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.126 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.128 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-04-22T01:15:58.130 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.131 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.132 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.134 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.135 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.136 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.138 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.143 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.144 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.145 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.146 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.147 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.149 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.150 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.153 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.154 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.157 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-04-22T01:15:58.158 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.160 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.162 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.164 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.165 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.166 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-04-22T01:15:58.169 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.170 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.171 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.173 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T01:15:58.175 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.176 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.177 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.178 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.179 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T01:15:58.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.182 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.183 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.185 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.186 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.187 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.188 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.190 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.191 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.192 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.193 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.195 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.197 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T01:15:58.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.199 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.200 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.201 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.205 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T01:15:58.206 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.208 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.209 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.210 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.212 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.213 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-04-22T01:15:58.214 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.215 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-04-22T01:15:58.220 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T01:15:58.221 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.222 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.224 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.225 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.226 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.227 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.228 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.229 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.230 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.232 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.233 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.234 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.235 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.239 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.241 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.242 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.246 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-04-22T01:15:58.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.250 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.251 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.252 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.254 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-04-22T01:15:58.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.256 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.261 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.262 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.263 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-04-22T01:15:58.266 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.267 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.268 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.269 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.271 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.274 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.275 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.276 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.279 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.282 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.284 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-04-22T01:15:58.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.287 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.289 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.290 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.291 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.293 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.294 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.295 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.296 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.298 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.302 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.303 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.305 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.306 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.308 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.309 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.310 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-04-22T01:15:58.311 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.312 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.313 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.314 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.315 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.317 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.318 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.319 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.320 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-04-22T01:15:58.326 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.327 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T01:15:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.329 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T01:15:58.330 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T01:15:58.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.332 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T01:15:58.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "services": { 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "dashboard": "https://172.21.15.38:8443/" 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "always_on_modules": { 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus": [ 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T01:15:58.337 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific": [ 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T01:15:58.338 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy": [ 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.339 INFO:teuthology.orchestra.run.smithi038.stdout: "reef": [ 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "squid": [ 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T01:15:58.340 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "last_failure_osd_epoch": 3, 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "active_clients": [ 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 486040671 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "libcephsqlite", 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 84523250 2024-04-22T01:15:58.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2298360886 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T01:15:58.343 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1392401979 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T01:15:58.344 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:15:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f522c0a2b90 msgr2=0x7f522c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f522c0a2b90 0x7f522c0a5050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f52441016f0 tx=0x7f5234005f90 comp rx=0 tx=0).stop 2024-04-22T01:15:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 msgr2=0x7f524419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:15:58.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f524419eff0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f523c02c700 tx=0x7f523c032f20 comp rx=0 tx=0).stop 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 shutdown_connections 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f522c0a2b90 0x7f522c0a5050 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5244102250 0x7f524419eff0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5244101890 0x7f5244197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 --2- 172.21.15.38:0/1537214883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52441006a0 0x7f5244197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 >> 172.21.15.38:0/1537214883 conn(0x7f52440fbc30 msgr2=0x7f5244104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 shutdown_connections 2024-04-22T01:15:58.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:15:57.115+0000 7f52499d7640 1 -- 172.21.15.38:0/1537214883 wait complete. 2024-04-22T01:15:58.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/3051375674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:56 smithi038 ceph-mon[28043]: pgmap v463: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/743910278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2701131514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:56 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/402496935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/3051375674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:56 smithi203 ceph-mon[31825]: pgmap v463: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/743910278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2701131514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:56 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/402496935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:57 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1537214883' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/3051375674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:56 smithi111 ceph-mon[31057]: pgmap v463: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-22T01:15:58.347 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/743910278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.348 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2701131514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.348 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:56 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/402496935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:15:58.348 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:57 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1537214883' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T01:15:58.354 DEBUG:tasks.check_counter:Getting stats from a 2024-04-22T01:15:58.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-22T01:15:58.355 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:57 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1537214883' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T01:15:58.616 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:15:59.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:15:58 smithi111 ceph-mon[31057]: pgmap v464: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-22T01:15:59.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:15:58 smithi038 ceph-mon[28043]: pgmap v464: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-22T01:15:59.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:15:58 smithi203 ceph-mon[31825]: pgmap v464: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-22T01:16:01.082 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 413, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 405, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 847920, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 154214, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.054064296, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.016865529, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.025007038, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.012649420, 2024-04-22T01:16:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 405, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.032489187, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000080220 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 746784, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 58496 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 229, 2024-04-22T01:16:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 299, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 56900, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 268924, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.028247529, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.009975853, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.006795359, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.008919014, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 297, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.013269015, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000044676 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 196352 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 307, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 309, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 82047, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 97627, 2024-04-22T01:16:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.027108546, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.008451127, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.007796963, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.008187555, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 307, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.013612740, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000044341 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1380, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.042488813, 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000030788 2024-04-22T01:16:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 0, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 0, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 0, 2024-04-22T01:16:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 1, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-22T01:16:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 0 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-22T01:16:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 6, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 2, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4194304, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4195956, 2024-04-22T01:16:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 4195815, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 6 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 3, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 3, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 0, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 1, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 1, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 0, 2024-04-22T01:16:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 49748, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 215420 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 0, 2024-04-22T01:16:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-22T01:16:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-22T01:16:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 0, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 2, 2024-04-22T01:16:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 1, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 1349, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 189319, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 50, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 175, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T01:16:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 8464, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 13, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 836, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 21, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 3, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 836, 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-22T01:16:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 836, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.534990367, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000639940 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 836, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 837 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 836, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 836, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 0, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 526, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 0, 2024-04-22T01:16:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 308, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 1, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-22T01:16:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 93, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 4, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 5, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 5, 2024-04-22T01:16:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 0, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 2, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 0 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 0, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 945, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 893306, 2024-04-22T01:16:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 945, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 945, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 893306, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 836, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 1657, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 836, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1657, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 836, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 836, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 836, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 836, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T01:16:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x5567733846e0": { 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:01.126 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:16:01.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:00 smithi203 ceph-mon[31825]: pgmap v465: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-22T01:16:01.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:00 smithi038 ceph-mon[28043]: pgmap v465: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-22T01:16:01.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:00 smithi111 ceph-mon[31057]: pgmap v465: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-22T01:16:01.943 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-22T01:16:01.943 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-22T01:16:01.944 DEBUG:tasks.check_counter:Getting stats from d 2024-04-22T01:16:01.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-22T01:16:02.202 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:16:03.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:02 smithi203 ceph-mon[31825]: pgmap v466: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:03.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:02 smithi111 ceph-mon[31057]: pgmap v466: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:03.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:02 smithi038 ceph-mon[28043]: pgmap v466: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-22T01:16:04.556 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001308069, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000263690, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 5, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 78, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3184, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 91440, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 3, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.007095891, 2024-04-22T01:16:04.558 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.004622666, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001624531, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 77, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.004345194, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000056431 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 91440 2024-04-22T01:16:04.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 106, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 101, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 721504, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 60848, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.022097060, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.006969607, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.017013854, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.002135541, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 100, 2024-04-22T01:16:04.560 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.005427376, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000054273 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 721504, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 60848 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T01:16:04.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:04.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:04.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148840, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 28, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T01:16:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T01:16:04.565 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 107, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 706597, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 107, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 107, 2024-04-22T01:16:04.566 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 706597, 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:04.567 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:16:05.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:04 smithi203 ceph-mon[31825]: pgmap v467: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:05.277 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:04 smithi038 ceph-mon[28043]: pgmap v467: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:05.309 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-22T01:16:05.309 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-22T01:16:05.309 DEBUG:tasks.check_counter:Getting stats from g 2024-04-22T01:16:05.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-22T01:16:05.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:04 smithi111 ceph-mon[31057]: pgmap v467: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:16:07.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:06 smithi038 ceph-mon[28043]: pgmap v468: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:07.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:06 smithi203 ceph-mon[31825]: pgmap v468: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:07.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:06 smithi111 ceph-mon[31057]: pgmap v468: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 311, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 310, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 106187, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 97757, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.037235203, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.014809422, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.008532250, 2024-04-22T01:16:07.544 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.009613692, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 309, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.017913508, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000057972 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.545 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 308, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 380, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 81214, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 297619, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.029839226, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.011239716, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.007907886, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.007854271, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 378, 2024-04-22T01:16:07.546 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.012333000, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000032626 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 199936 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 113, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 107, 2024-04-22T01:16:07.547 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 745652, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 60293, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.020785830, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.006348774, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.016669118, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.002056382, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 106, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.006964680, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000065704 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:07.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 744304, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 59328 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T01:16:07.549 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1225, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.033686033, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000027498 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.550 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 0, 2024-04-22T01:16:07.551 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 1, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:07.554 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 0 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:07.555 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-22T01:16:07.556 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 2, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 2, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-22T01:16:07.557 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4194304, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4194892, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 4194352, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 2 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 3, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 3, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 0, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 1, 2024-04-22T01:16:07.558 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 1, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 50320, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 215420 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 0, 2024-04-22T01:16:07.559 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.560 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.561 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.562 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-22T01:16:07.563 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.564 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.567 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-22T01:16:07.570 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 197336, 2024-04-22T01:16:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 53, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 175, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 8464, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 13, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 616, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 21, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 3, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 616, 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-22T01:16:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 616, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 2.302958511, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.003738569 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 616, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 617 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 616, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 616, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 0, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-22T01:16:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 305, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 309, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 1, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 0, 2024-04-22T01:16:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-22T01:16:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 93, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 4, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 6, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 6, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-22T01:16:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 0, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 2, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 0 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 0, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 728, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 858918, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:07.581 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 728, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 728, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 858918, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:07.582 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 616, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 593, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 616, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 593, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:07.583 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 616, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 616, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 616, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 616, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.584 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x55672a8bc6e0": { 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:07.586 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:07.587 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:07.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:07.587 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:16:08.604 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-22T01:16:08.605 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-22T01:16:08.605 DEBUG:tasks.check_counter:Getting stats from j 2024-04-22T01:16:08.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-22T01:16:08.854 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T01:16:09.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:08 smithi038 ceph-mon[28043]: pgmap v469: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-22T01:16:09.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:08 smithi203 ceph-mon[31825]: pgmap v469: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-22T01:16:09.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:08 smithi111 ceph-mon[31057]: pgmap v469: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-22T01:16:10.709 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 4, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 4, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3024, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 448, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001590726, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000043213, 2024-04-22T01:16:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000589829, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000814031, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000271343 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T01:16:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 1, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 78, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 160, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 91344, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 3, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.007354082, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.004751533, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001802817, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 76, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.003482597, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000045823 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 91344 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 108, 2024-04-22T01:16:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 103, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 722048, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 62128, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.023155766, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.007041004, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.017777350, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.002443321, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 102, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.013205414, 2024-04-22T01:16:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000129464 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 722048, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 62128 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148840, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 28, 2024-04-22T01:16:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 109, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 706857, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 109, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 109, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 706857, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T01:16:10.721 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T01:16:11.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:10 smithi203 ceph-mon[31825]: pgmap v470: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T01:16:11.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:10 smithi111 ceph-mon[31057]: pgmap v470: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T01:16:11.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:10 smithi038 ceph-mon[28043]: pgmap v470: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T01:16:11.566 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-22T01:16:11.566 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-22T01:16:11.566 DEBUG:tasks.check_counter:Getting stats from b 2024-04-22T01:16:11.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-22T01:16:11.814 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:16:13.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:12 smithi111 ceph-mon[31057]: pgmap v471: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-22T01:16:13.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:12 smithi203 ceph-mon[31825]: pgmap v471: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-22T01:16:13.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:12 smithi038 ceph-mon[28043]: pgmap v471: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-22T01:16:13.817 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6555, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 6482, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1686152, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 10797695, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 37, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.515508444, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.168090660, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.125670056, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.171915814, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 6483, 2024-04-22T01:16:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.503625165, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000077683 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 394, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000076210, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000193 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 1887, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 1791, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 486523, 2024-04-22T01:16:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 1247147, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 9, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 7, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.141522000, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.042795118, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.040003347, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.047710333, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1789, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.187009091, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000104532 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 582, 2024-04-22T01:16:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000109837, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000188 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 212560 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6729, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 7805, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 3028131, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 3507602, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 53, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 7, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.529668721, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.200905743, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.383720434, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.077801233, 2024-04-22T01:16:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 7802, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.840190745, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000107689 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 8507, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002179302, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000256 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 2467728, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 2334304 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T01:16:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4976, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2.683967847, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000539382 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4306, 2024-04-22T01:16:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.139606098, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000032421 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 64, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 64, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 64, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.075814707, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001184604 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-22T01:16:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 75, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 25, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 5, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 20, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 17, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 2, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 2, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 2, 2024-04-22T01:16:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 96, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 87, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 64, 2024-04-22T01:16:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 4, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 234, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 64, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 42, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 5, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 37, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 22, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 1, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 12, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 1468, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 4087, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 4071, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 4, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 26 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 7, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 5, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 5, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 5, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 4173, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 4162, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 4162, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 11, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 7, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 17, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 16, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 16, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 1, 2024-04-22T01:16:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 1, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 1, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 13005265, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 13033944, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4151, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 81.325720225, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.019591838 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 23, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 33, 2024-04-22T01:16:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 10, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 18, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 20, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 2, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 25, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 33, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 8, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 2, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 20, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 18, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 57952, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 231804 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 84, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 1, 2024-04-22T01:16:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 64, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 20, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 2, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 7, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.007490288, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001070041 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 27, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.007162699, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000265285 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000363445, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000363445 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 8, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.015051513, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001881439 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000251109, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000251109 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.006431312, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003215656 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002084675, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001042337 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.008808095, 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004404047 2024-04-22T01:16:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001635893, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000817946 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9, 2024-04-22T01:16:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.020008306, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002223145 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.006527372, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002175790 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 1, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 4, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 3, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 1, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 11, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 11, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 189, 2024-04-22T01:16:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 315571, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 132, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 440, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 5, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 175, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 95258, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 635, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T01:16:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 9581, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 8927932, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 32, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 9581, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9581, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 136.990582409, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.014298150 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9581, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 9810 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 9581, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 79, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 9502, 2024-04-22T01:16:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 55, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 41, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 10, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 4156, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 53, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 5131, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 69, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 75, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T01:16:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 93, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 9, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 12, 2024-04-22T01:16:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 12, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 203, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 17 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T01:16:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 3585, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 2, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 4615, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 5, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 15167, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 3634011, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 15167, 2024-04-22T01:16:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 15167, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 3634011, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 9581, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 8928461, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 9581, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 8928461, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T01:16:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 9581, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 9581, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 9581, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 9581, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 5, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 529, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 5, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 5, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 529, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x55fc4fed6500": { 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 4173, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 8839640, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 4173, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 4151, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 8839640, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:13.859 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T01:16:14.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:13 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:16:14.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:13 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:16:14.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:13 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T01:16:14.724 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-22T01:16:14.724 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-22T01:16:14.724 DEBUG:tasks.check_counter:Getting stats from e 2024-04-22T01:16:14.724 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-22T01:16:14.970 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:16:15.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:14 smithi111 ceph-mon[31057]: pgmap v472: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-22T01:16:15.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:16:15.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:16:15.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:14 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:16:15.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:14 smithi203 ceph-mon[31825]: pgmap v472: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-22T01:16:15.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:16:15.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:16:15.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:14 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:16:15.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:14 smithi038 ceph-mon[28043]: pgmap v472: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-22T01:16:15.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T01:16:15.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T01:16:15.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:14 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 173, 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 170, 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 760547, 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 238684, 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 21, 2024-04-22T01:16:16.735 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.030984160, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.008476011, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.022597303, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.002762815, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 165, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.017286977, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000104769 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 14, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000001353, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000096 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.736 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 756064, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 69104 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 117, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 273, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 34417, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 249756, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 7, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.025652001, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.012702721, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.008652677, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.002025874, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 276, 2024-04-22T01:16:16.737 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.026626978, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000096474 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 286, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000086436, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000302 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 204496 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 17, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 19, 2024-04-22T01:16:16.738 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 4573, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 8091, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 6, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.002037050, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000129919, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.001606494, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000214611, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 17, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.018304068, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001076709 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:16.739 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 35, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001341679, 2024-04-22T01:16:16.740 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000038333 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000015767, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000007883 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 0, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 0, 2024-04-22T01:16:16.741 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 11, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 10, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 0, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 10, 2024-04-22T01:16:16.742 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 10, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 1, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-22T01:16:16.743 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 5, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 10737418474, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:16.744 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-22T01:16:16.745 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 1, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:16.746 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 2, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 2, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T01:16:16.747 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 2, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4194892, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001747544, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001747544 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-22T01:16:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 13, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 13, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 0, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 11, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 12, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 1, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 10, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 10, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 0, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 0, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 0, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 51232, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 223612 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.749 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.750 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.751 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.752 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T01:16:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.754 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T01:16:16.755 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.757 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.758 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.759 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 0, 2024-04-22T01:16:16.761 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:16.762 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:16.763 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 229831, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 69, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T01:16:16.764 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 175, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 50664, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 198, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:16.765 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 35, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 10710, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 33, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 35, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 35, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.214996201, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.006142748 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T01:16:16.766 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 35, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 57 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 35, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 35, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 10, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 1, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 5, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T01:16:16.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 6, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 20, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:16.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T01:16:16.769 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 93, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 4, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 9, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 9, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 14, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 1 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:16.770 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:16.771 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 303, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 761775, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 303, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 303, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 761775, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.772 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 35, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 10710, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 35, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 10710, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 35, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 35, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 35, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 35, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.774 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T01:16:16.775 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x558669ac0500": { 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 2, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 588, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 2, 2024-04-22T01:16:16.776 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 1, 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 588, 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:16.777 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T01:16:17.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:16 smithi203 ceph-mon[31825]: pgmap v473: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-22T01:16:17.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:16 smithi111 ceph-mon[31057]: pgmap v473: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-22T01:16:17.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:16 smithi038 ceph-mon[28043]: pgmap v473: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-22T01:16:17.752 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-22T01:16:17.752 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-22T01:16:17.752 DEBUG:tasks.check_counter:Getting stats from h 2024-04-22T01:16:17.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-22T01:16:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:16:19.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:19 smithi111 ceph-mon[31057]: pgmap v474: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-22T01:16:19.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:19 smithi203 ceph-mon[31825]: pgmap v474: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-22T01:16:19.302 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:19 smithi038 ceph-mon[28043]: pgmap v474: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-22T01:16:19.785 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 181, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 257, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 54394, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 165792, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 12, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.031026579, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.012219122, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.013359048, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.002523068, 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:19.786 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 255, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.039437817, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000154658 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 369, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000083915, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000227 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 36, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 111, 2024-04-22T01:16:19.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 8491, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 331499, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 19, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 7, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.015773619, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.004514776, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.007848874, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000415254, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 109, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.023013077, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000211129 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:19.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 38, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000004303, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000113 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 4256, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 208608 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 172, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 157, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 777931, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 79396, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 6, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.027481123, 2024-04-22T01:16:19.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.007863262, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.020955063, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.003024747, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 157, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.018494323, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000117798 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 7, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000001351, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000193 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 775120, 2024-04-22T01:16:19.793 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 69392 2024-04-22T01:16:19.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T01:16:19.793 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T01:16:19.793 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 53, 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002324192, 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000043852 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:19.794 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000008045, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000004022 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 8, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 5, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002449017, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000489803 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 3, 2024-04-22T01:16:19.795 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 11, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 12, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 12, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 11, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 1, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 1, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 2, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 11, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 8, 2024-04-22T01:16:19.796 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 2, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 5, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 10737418474, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-22T01:16:19.797 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:19.798 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 1 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 3, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-22T01:16:19.799 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.800 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 7, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 7, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 2, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-22T01:16:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4196264, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 6, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.013706354, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002284392 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T01:16:19.802 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 15, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 15, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 0, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 13, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 14, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 1, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 12, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 12, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 0, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 1, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 5, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 4, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 51484, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 223612 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.803 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 13, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 8, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 11, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 1, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001820164, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000910082 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000515101, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000257550 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000113752, 2024-04-22T01:16:19.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000113752 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T01:16:19.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T01:16:19.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:19.814 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 1, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 3, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 2, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 1, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 3, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 1, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 184, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.815 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:19.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T01:16:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 248327, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 96, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 176, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 2, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:19.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 175, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 59480, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 238, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T01:16:19.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 45, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 12577, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 33, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 45, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 45, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.248995601, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005533235 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 45, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 69 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 45, 2024-04-22T01:16:19.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 45, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 10, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 6, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 8, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 6, 2024-04-22T01:16:19.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 20, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T01:16:19.822 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 93, 2024-04-22T01:16:19.823 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 4, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 9, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 9, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 17, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 2 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:19.825 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 385, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 794661, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 385, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 385, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 794661, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.826 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 45, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 12577, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 45, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 12577, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 45, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 45, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 45, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 45, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.828 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.829 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x56488d45a500": { 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 7, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1960, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 7, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 6, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1960, 2024-04-22T01:16:19.830 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:19.831 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:19.831 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:19.831 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:19.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:19.831 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T01:16:20.593 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-22T01:16:20.593 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-22T01:16:20.594 DEBUG:tasks.check_counter:Getting stats from k 2024-04-22T01:16:20.594 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-22T01:16:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T01:16:21.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:21 smithi203 ceph-mon[31825]: pgmap v475: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:16:21.286 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:21 smithi111 ceph-mon[31057]: pgmap v475: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:16:21.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:21 smithi038 ceph-mon[28043]: pgmap v475: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:16:22.623 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T01:16:22.623 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:22.623 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 448, 2024-04-22T01:16:22.623 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 440, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 859472, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 169068, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 1, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.058253375, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.021424601, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.023502697, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.012897454, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 440, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.022548917, 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000051247 2024-04-22T01:16:22.624 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 751280, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 66672 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 3, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 82, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 716, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 205098, 2024-04-22T01:16:22.625 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.005225220, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.003426408, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.000940470, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000050629, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 80, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.004767729, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000059596 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:22.626 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 204400 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 576, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 578, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 149453, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 183175, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:22.627 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.037432932, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.008763453, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.010627248, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.015590614, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 576, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.016995290, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000029505 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.628 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1541, 2024-04-22T01:16:22.629 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.039720200, 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000025775 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T01:16:22.630 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T01:16:22.631 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 1, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:22.632 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 0, 2024-04-22T01:16:22.633 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:22.634 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T01:16:22.635 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:22.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 7, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 0, 2024-04-22T01:16:22.637 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4196264, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 4196123, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 7 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 3, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 3, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 0, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 1, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 1, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 0, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 0, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 0, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 0, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-22T01:16:22.639 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 49780, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 215420 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.641 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.642 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.643 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.644 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.648 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T01:16:22.649 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 1, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 3, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 2, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 2, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 0, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 2604, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:22.652 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 181199, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 45, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:22.655 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 175, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 8464, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 13, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T01:16:22.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 900, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 21, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 3, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 900, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 900, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.565990005, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000628877 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T01:16:22.657 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 900, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 901 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 900, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 900, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 568, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 330, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T01:16:22.658 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 1, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T01:16:22.659 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T01:16:22.661 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:22.661 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T01:16:22.662 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:22.662 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T01:16:22.662 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T01:16:22.662 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T01:16:22.662 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T01:16:22.662 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T01:16:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 93, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 4, 2024-04-22T01:16:22.664 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 4, 2024-04-22T01:16:22.665 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 4, 2024-04-22T01:16:22.665 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T01:16:22.665 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T01:16:22.665 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 0, 2024-04-22T01:16:22.665 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 2, 2024-04-22T01:16:22.665 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 0 2024-04-22T01:16:22.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.666 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T01:16:22.666 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 0, 2024-04-22T01:16:22.666 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:22.666 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T01:16:22.666 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:22.667 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 1023, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 908345, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 1023, 2024-04-22T01:16:22.668 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 1023, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 908345, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:22.669 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 900, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 1965, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 900, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1965, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:22.670 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 900, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 900, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 900, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 900, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T01:16:22.672 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x55b2623cc6e0": { 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T01:16:22.673 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T01:16:22.674 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T01:16:23.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:23 smithi203 ceph-mon[31825]: pgmap v476: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:23.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:23 smithi111 ceph-mon[31057]: pgmap v476: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:23.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:23 smithi038 ceph-mon[28043]: pgmap v476: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:23.556 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-22T01:16:23.556 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-22T01:16:23.556 DEBUG:tasks.check_counter:Getting stats from c 2024-04-22T01:16:23.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-22T01:16:23.806 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:16:25.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:25 smithi203 ceph-mon[31825]: pgmap v477: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:25.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:25 smithi111 ceph-mon[31057]: pgmap v477: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:25.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:25 smithi038 ceph-mon[28043]: pgmap v477: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:25.749 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T01:16:25.749 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:25.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 186, 2024-04-22T01:16:25.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 177, 2024-04-22T01:16:25.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 770026, 2024-04-22T01:16:25.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 196675, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 18, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.032432241, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.008843123, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.022429198, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.003166870, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 176, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.016969461, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000096417 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 24, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000003693, 2024-04-22T01:16:25.750 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000153 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 764352, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 70576 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 17, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 94, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 4029, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 258225, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 10, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.008603652, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.003773838, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.003444784, 2024-04-22T01:16:25.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000443493, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 92, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.014538000, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000158021 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000534, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000133 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 205664 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:25.752 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 130, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 218, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 43913, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 78320, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.022588345, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.008392673, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.009845761, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.002627903, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 217, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.020829811, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000095989 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.753 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 307, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000093545, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000304 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:25.754 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 36, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002139105, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000059419 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000009598, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000004799 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T01:16:25.755 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 10, 2024-04-22T01:16:25.756 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 10, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 10, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T01:16:25.757 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 5, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 10737418474, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:25.758 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-22T01:16:25.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 1, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:25.760 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 6, 2024-04-22T01:16:25.761 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 6, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4195956, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-04-22T01:16:25.762 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.012646128, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002529225 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 13, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 13, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 0, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 11, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 12, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 1, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 10, 2024-04-22T01:16:25.763 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 10, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 50760, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 8, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.764 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T01:16:25.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:25.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T01:16:25.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.770 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.771 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.772 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.773 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T01:16:25.774 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 0, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-22T01:16:25.775 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 2, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 68, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:25.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:25.777 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 222023, 2024-04-22T01:16:25.778 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 69, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 175, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T01:16:25.779 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 50664, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 198, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 42, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 12044, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 42, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 42, 2024-04-22T01:16:25.780 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.246993872, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005880806 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 42, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 64 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 42, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 0, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 42, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-04-22T01:16:25.781 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 5, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 8, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 6, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:25.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T01:16:25.783 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 93, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 4, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 8, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 8, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T01:16:25.784 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 14, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 1 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:25.785 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 329, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 777436, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 329, 2024-04-22T01:16:25.786 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 329, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 777436, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:25.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 42, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 12044, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 42, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 12044, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:25.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 42, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 42, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 42, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 42, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:25.789 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.790 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55915c67e500": { 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 6, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1652, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 6, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 5, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1652, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:25.791 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:25.792 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:25.792 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.792 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:25.792 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T01:16:26.606 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-22T01:16:26.606 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-22T01:16:26.606 DEBUG:tasks.check_counter:Getting stats from f 2024-04-22T01:16:26.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-22T01:16:26.856 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:16:27.184 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:27 smithi203 ceph-mon[31825]: pgmap v478: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:27.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:27 smithi111 ceph-mon[31057]: pgmap v478: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:27.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:27 smithi038 ceph-mon[28043]: pgmap v478: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 620, 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 620, 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 4382973, 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 196346, 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-22T01:16:28.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.069827883, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.019715222, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.022401496, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.021218889, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 619, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.027099239, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000043779 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 162, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 240, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 492638, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 257104, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.017931996, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.007487475, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.004587409, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004116733, 2024-04-22T01:16:28.808 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 238, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.015360056, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000064538 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 205840 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 395, 2024-04-22T01:16:28.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 389, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 5009987, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 152237, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 3, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.051813929, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.016153594, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.025326259, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.012108202, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 388, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.017438221, 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000044943 2024-04-22T01:16:28.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 747872, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 65104 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T01:16:28.811 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1739, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.052679524, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000030292 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.812 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T01:16:28.813 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 22, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 6, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 16, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 13, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 2, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-22T01:16:28.814 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-22T01:16:28.815 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-22T01:16:28.816 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-22T01:16:28.817 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T01:16:28.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 4173, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 13, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T01:16:28.819 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 13005265, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 13033944, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 13030186, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 4173 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 17, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 22, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 5, 2024-04-22T01:16:28.820 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 12, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 14, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 2, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 22, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 26, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 4, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 52664, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-22T01:16:28.821 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.822 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.825 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.830 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T01:16:28.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 4, 2024-04-22T01:16:28.832 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 3, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2715, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:28.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:28.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:28.835 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 189256, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 49, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 175, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T01:16:28.836 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 68746, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 315, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 1051, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 26, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 4, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 1051, 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T01:16:28.837 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1051, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.719982216, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000685044 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1051, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1052 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 1051, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 1051, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 656, 2024-04-22T01:16:28.838 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 392, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 2, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:28.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T01:16:28.840 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 93, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 4, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 5, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 5, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T01:16:28.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T01:16:28.842 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1173, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 9770860, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:28.843 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1173, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1173, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 9770860, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:28.844 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1051, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 8839650, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1051, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 8839650, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.845 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1051, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 1051, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1051, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1051, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.846 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T01:16:28.847 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55e5f09b46e0": { 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:28.848 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:28.849 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T01:16:29.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:29 smithi111 ceph-mon[31057]: pgmap v479: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T01:16:29.399 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:29 smithi203 ceph-mon[31825]: pgmap v479: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T01:16:29.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:29 smithi038 ceph-mon[28043]: pgmap v479: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T01:16:29.635 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-22T01:16:29.635 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-22T01:16:29.635 DEBUG:tasks.check_counter:Getting stats from i 2024-04-22T01:16:29.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-22T01:16:29.874 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:16:31.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:31 smithi203 ceph-mon[31825]: pgmap v480: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:16:31.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:31 smithi111 ceph-mon[31057]: pgmap v480: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:16:31.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:31 smithi038 ceph-mon[28043]: pgmap v480: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 36, 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 39, 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 11250, 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 62001, 2024-04-22T01:16:31.599 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 12, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.008896724, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.001343099, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.004916375, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000525212, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 38, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.014915284, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000392507 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 18, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000002630, 2024-04-22T01:16:31.600 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000146 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 5088, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 3760 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 27, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 104, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 6329, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 293196, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 13, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.010004672, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.003984299, 2024-04-22T01:16:31.601 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.004525026, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000379311, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 102, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.019237633, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000188604 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 13, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000002066, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000158 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 209632 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.602 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 351, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 387, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 828524, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 213868, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 9, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.048262966, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.017322411, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.028575803, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.005084046, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 386, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.022005689, 2024-04-22T01:16:31.603 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000057009 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 419, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000117891, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000281 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 767664, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 69424 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.604 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 53, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001478105, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000027888 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000011504, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000005752 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.605 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 2, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 1, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001107394, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001107394 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 1, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T01:16:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 14, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 14, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 14, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 1, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 1, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 2, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 3, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 1, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 1, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:31.607 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 5, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 10737418474, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:31.608 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 2, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 3, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 2, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:31.609 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 1 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 2, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T01:16:31.610 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T01:16:31.611 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 5, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 5, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T01:16:31.612 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4195732, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.007523409, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001880852 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 17, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 17, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 0, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 15, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 16, 2024-04-22T01:16:31.613 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 1, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 14, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 14, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 0, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 1, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 1, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 50812, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 3, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 2, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 7, 2024-04-22T01:16:31.614 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001107394, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001107394 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.616 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.619 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T01:16:31.619 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.619 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.619 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.621 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.622 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.623 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.624 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:31.625 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 1, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 1, 2024-04-22T01:16:31.626 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 206, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:31.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:31.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 238535, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 76, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:31.629 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 175, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 67765, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 288, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:31.630 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 42, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 12135, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 31, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 42, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 42, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.234994180, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005595099 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T01:16:31.631 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 42, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 66 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 42, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 42, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 4, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 7, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T01:16:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 6, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:31.633 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T01:16:31.634 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 93, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 4, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 8, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 8, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 18, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 1 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T01:16:31.635 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-22T01:16:31.636 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 410, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 798088, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 410, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 410, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 798088, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.637 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 42, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 12135, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 42, 2024-04-22T01:16:31.638 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 12135, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:31.639 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 42, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 42, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 42, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 42, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:31.640 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55fa44a62500": { 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:31.641 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 5, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1428, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 5, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 4, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1428, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:31.642 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T01:16:32.388 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-22T01:16:32.388 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-22T01:16:32.389 DEBUG:tasks.check_counter:Getting stats from l 2024-04-22T01:16:32.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid caf31b2e-0043-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-22T01:16:32.629 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T01:16:33.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:33 smithi203 ceph-mon[31825]: pgmap v481: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:33.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:33 smithi111 ceph-mon[31057]: pgmap v481: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:33.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:33 smithi038 ceph-mon[28043]: pgmap v481: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T01:16:34.332 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T01:16:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T01:16:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 126, 2024-04-22T01:16:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 120, 2024-04-22T01:16:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 749906, 2024-04-22T01:16:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 68468, 2024-04-22T01:16:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 1, 2024-04-22T01:16:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.023608369, 2024-04-22T01:16:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.006726068, 2024-04-22T01:16:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.018283279, 2024-04-22T01:16:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.002787356, 2024-04-22T01:16:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 119, 2024-04-22T01:16:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.005980023, 2024-04-22T01:16:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000050252 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 749072, 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 68144 2024-04-22T01:16:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T01:16:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 213, 2024-04-22T01:16:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 295, 2024-04-22T01:16:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 52334, 2024-04-22T01:16:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 275581, 2024-04-22T01:16:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-22T01:16:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-22T01:16:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.028242497, 2024-04-22T01:16:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.009380594, 2024-04-22T01:16:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.006651947, 2024-04-22T01:16:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.009086829, 2024-04-22T01:16:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 293, 2024-04-22T01:16:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.015711974, 2024-04-22T01:16:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000053624 2024-04-22T01:16:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T01:16:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 208160 2024-04-22T01:16:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T01:16:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 672, 2024-04-22T01:16:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 672, 2024-04-22T01:16:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 203965, 2024-04-22T01:16:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 212930, 2024-04-22T01:16:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-04-22T01:16:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-22T01:16:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.070839529, 2024-04-22T01:16:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.021772424, 2024-04-22T01:16:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.019671096, 2024-04-22T01:16:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.023119551, 2024-04-22T01:16:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T01:16:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 671, 2024-04-22T01:16:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.034901872, 2024-04-22T01:16:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000052014 2024-04-22T01:16:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T01:16:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-22T01:16:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T01:16:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T01:16:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T01:16:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T01:16:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T01:16:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1514, 2024-04-22T01:16:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.047377075, 2024-04-22T01:16:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000031292 2024-04-22T01:16:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T01:16:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T01:16:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T01:16:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T01:16:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-22T01:16:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-22T01:16:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T01:16:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T01:16:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-22T01:16:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T01:16:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T01:16:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-22T01:16:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T01:16:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T01:16:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 0, 2024-04-22T01:16:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-04-22T01:16:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 0, 2024-04-22T01:16:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-22T01:16:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 0, 2024-04-22T01:16:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-22T01:16:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-22T01:16:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-22T01:16:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-04-22T01:16:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T01:16:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-22T01:16:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-22T01:16:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T01:16:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-22T01:16:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T01:16:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-04-22T01:16:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-04-22T01:16:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T01:16:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T01:16:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T01:16:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-22T01:16:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T01:16:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T01:16:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-22T01:16:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T01:16:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T01:16:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T01:16:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-22T01:16:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T01:16:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-22T01:16:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-22T01:16:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-22T01:16:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T01:16:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T01:16:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T01:16:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T01:16:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T01:16:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T01:16:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-22T01:16:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T01:16:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T01:16:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T01:16:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T01:16:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T01:16:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 5, 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T01:16:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T01:16:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4195732, 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 4195591, 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T01:16:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 5 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T01:16:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 3, 2024-04-22T01:16:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 3, 2024-04-22T01:16:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 0, 2024-04-22T01:16:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 1, 2024-04-22T01:16:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 1, 2024-04-22T01:16:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 0, 2024-04-22T01:16:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 0, 2024-04-22T01:16:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 0, 2024-04-22T01:16:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 0, 2024-04-22T01:16:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-22T01:16:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-22T01:16:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-22T01:16:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 49672, 2024-04-22T01:16:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-04-22T01:16:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T01:16:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-22T01:16:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-22T01:16:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-22T01:16:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-22T01:16:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-22T01:16:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T01:16:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T01:16:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T01:16:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T01:16:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T01:16:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T01:16:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T01:16:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T01:16:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T01:16:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T01:16:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T01:16:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T01:16:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T01:16:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T01:16:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T01:16:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T01:16:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T01:16:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T01:16:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T01:16:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T01:16:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T01:16:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T01:16:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T01:16:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T01:16:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T01:16:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T01:16:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T01:16:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T01:16:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T01:16:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2721, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T01:16:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T01:16:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T01:16:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 190040, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 51, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9088, 2024-04-22T01:16:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 175, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 8464, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 13, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T01:16:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 881, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 881, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 881, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 3.394916161, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.003853480 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 881, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 882 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 881, 2024-04-22T01:16:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 881, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 541, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 338, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T01:16:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T01:16:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 93, 2024-04-22T01:16:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 4, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 5, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 5, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1007, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 906193, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1007, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1007, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 906193, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 881, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 1433, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 881, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1433, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 881, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 881, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 881, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 881, 2024-04-22T01:16:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x563089dfc6e0": { 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T01:16:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T01:16:34.414 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T01:16:35.113 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:35 smithi203 ceph-mon[31825]: pgmap v482: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:35.140 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-22T01:16:35.140 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-22T01:16:35.140 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-22T01:16:35.153 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T01:16:35.153 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:16:35.153 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:35.174 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T01:16:35.174 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T01:16:35.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:35.302 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:35 smithi038 ceph-mon[28043]: pgmap v482: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:35.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 -- 172.21.15.38:0/168697283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 msgr2=0x7efcf4102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 --2- 172.21.15.38:0/168697283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf4102e20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7efce4009900 tx=0x7efce402bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 -- 172.21.15.38:0/168697283 shutdown_connections 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 --2- 172.21.15.38:0/168697283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 0x7efcf4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 --2- 172.21.15.38:0/168697283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf4102e20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 --2- 172.21.15.38:0/168697283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf41017b0 0x7efcf4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 -- 172.21.15.38:0/168697283 >> 172.21.15.38:0/168697283 conn(0x7efcf40fcd40 msgr2=0x7efcf40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 -- 172.21.15.38:0/168697283 shutdown_connections 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.391+0000 7efcfc5de640 1 -- 172.21.15.38:0/168697283 wait complete. 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 Processor -- start 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 -- start start 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf41017b0 0x7efcf4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 0x7efcf419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efcf410d4a0 con 0x7efcf41029a0 2024-04-22T01:16:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efcf410d320 con 0x7efcf4103360 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfc5de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efcf410d620 con 0x7efcf41017b0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcf9b79640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcf9b79640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf41980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45560/0 (socket says 172.21.15.38:45560) 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcf9b79640 1 -- 172.21.15.38:0/204390032 learned_addr learned my addr 172.21.15.38:0/204390032 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.392+0000 7efcfa37a640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf41017b0 0x7efcf4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcfab7b640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 0x7efcf419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcf9b79640 1 -- 172.21.15.38:0/204390032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf41017b0 msgr2=0x7efcf4197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcf9b79640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf41017b0 0x7efcf4197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcf9b79640 1 -- 172.21.15.38:0/204390032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 msgr2=0x7efcf419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcf9b79640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 0x7efcf419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcf9b79640 1 -- 172.21.15.38:0/204390032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf419f7b0 con 0x7efcf41029a0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcfab7b640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 0x7efcf419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcf9b79640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf41980d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7efce4009230 tx=0x7efce4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4041070 con 0x7efcf41029a0 2024-04-22T01:16:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf419fa40 con 0x7efcf41029a0 2024-04-22T01:16:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce40054e0 con 0x7efcf41029a0 2024-04-22T01:16:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4034440 con 0x7efcf41029a0 2024-04-22T01:16:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.393+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf419ff80 con 0x7efcf41029a0 2024-04-22T01:16:35.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:35 smithi111 ceph-mon[31057]: pgmap v482: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T01:16:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.394+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf4108ad0 con 0x7efcf41029a0 2024-04-22T01:16:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.395+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7efce4044070 con 0x7efcf41029a0 2024-04-22T01:16:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.396+0000 7efceb7fe640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efce00a2b90 0x7efce00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.396+0000 7efcfa37a640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efce00a2b90 0x7efce00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.396+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7efce40f14c0 con 0x7efcf41029a0 2024-04-22T01:16:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.396+0000 7efcfa37a640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efce00a2b90 0x7efce00a5050 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efcf4102800 tx=0x7efcf0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.400+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efce40be010 con 0x7efcf41029a0 2024-04-22T01:16:35.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.587+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efcf4062340 con 0x7efcf41029a0 2024-04-22T01:16:35.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.589+0000 7efceb7fe640 1 -- 172.21.15.38:0/204390032 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7efce40bd910 con 0x7efcf41029a0 2024-04-22T01:16:35.589 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:16:35.589 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efce00a2b90 msgr2=0x7efce00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efce00a2b90 0x7efce00a5050 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efcf4102800 tx=0x7efcf0009110 comp rx=0 tx=0).stop 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 msgr2=0x7efcf41980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf41980d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7efce4009230 tx=0x7efce4005dd0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 shutdown_connections 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efce00a2b90 0x7efce00a5050 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efcf4103360 0x7efcf419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efcf41029a0 0x7efcf41980d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 --2- 172.21.15.38:0/204390032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf41017b0 0x7efcf4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 >> 172.21.15.38:0/204390032 conn(0x7efcf40fcd40 msgr2=0x7efcf4105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:35.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 shutdown_connections 2024-04-22T01:16:35.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.591+0000 7efcfc5de640 1 -- 172.21.15.38:0/204390032 wait complete. 2024-04-22T01:16:35.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T01:05:44.235660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T01:10:28.471176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T01:10:29.346433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-22T01:10:33.063023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T01:16:35.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.748+0000 7fa2cd682640 1 -- 172.21.15.38:0/4099638063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c81006a0 msgr2=0x7fa2c8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.748+0000 7fa2cd682640 1 --2- 172.21.15.38:0/4099638063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c81006a0 0x7fa2c8100ac0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009900 tx=0x7fa2b802bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.748+0000 7fa2cd682640 1 -- 172.21.15.38:0/4099638063 shutdown_connections 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.748+0000 7fa2cd682640 1 --2- 172.21.15.38:0/4099638063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c8102250 0x7fa2c8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.748+0000 7fa2cd682640 1 --2- 172.21.15.38:0/4099638063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2c8101890 0x7fa2c8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.748+0000 7fa2cd682640 1 --2- 172.21.15.38:0/4099638063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c81006a0 0x7fa2c8100ac0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- 172.21.15.38:0/4099638063 >> 172.21.15.38:0/4099638063 conn(0x7fa2c80fbc30 msgr2=0x7fa2c80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- 172.21.15.38:0/4099638063 shutdown_connections 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- 172.21.15.38:0/4099638063 wait complete. 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 Processor -- start 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- start start 2024-04-22T01:16:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c81006a0 0x7fa2c806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2c8101890 0x7fa2c819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 0x7fa2c819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c810c2f0 con 0x7fa2c8102250 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c810c170 con 0x7fa2c81006a0 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.749+0000 7fa2cd682640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c810c470 con 0x7fa2c8101890 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 0x7fa2c819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 0x7fa2c819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45582/0 (socket says 172.21.15.38:45582) 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 -- 172.21.15.38:0/2452548946 learned_addr learned my addr 172.21.15.38:0/2452548946 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c67fc640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2c8101890 0x7fa2c819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c6ffd640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c81006a0 0x7fa2c806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 -- 172.21.15.38:0/2452548946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2c8101890 msgr2=0x7fa2c819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2c8101890 0x7fa2c819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 -- 172.21.15.38:0/2452548946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c81006a0 msgr2=0x7fa2c806a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c81006a0 0x7fa2c806a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 -- 172.21.15.38:0/2452548946 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c819f6e0 con 0x7fa2c8102250 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c6ffd640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c81006a0 0x7fa2c806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2c77fe640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 0x7fa2c819ee20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000eb10 tx=0x7fa2c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0004b20 con 0x7fa2c8102250 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2c0004cc0 con 0x7fa2c8102250 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c819f9d0 con 0x7fa2c8102250 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0010760 con 0x7fa2c8102250 2024-04-22T01:16:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.750+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c819fed0 con 0x7fa2c8102250 2024-04-22T01:16:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.751+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c8107ae0 con 0x7fa2c8102250 2024-04-22T01:16:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.752+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fa2c0005020 con 0x7fa2c8102250 2024-04-22T01:16:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.753+0000 7fa2affff640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2b00a2b90 0x7fa2b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.753+0000 7fa2c6ffd640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2b00a2b90 0x7fa2b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.753+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fa2c00cf9f0 con 0x7fa2c8102250 2024-04-22T01:16:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.754+0000 7fa2c6ffd640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2b00a2b90 0x7fa2b00a5050 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa2b802c7b0 tx=0x7fa2b8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:35.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.758+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa2c009c540 con 0x7fa2c8102250 2024-04-22T01:16:35.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.941+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2c80623e0 con 0x7fa2c8102250 2024-04-22T01:16:35.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.941+0000 7fa2affff640 1 -- 172.21.15.38:0/2452548946 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fa2c009be40 con 0x7fa2c8102250 2024-04-22T01:16:35.942 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2b00a2b90 msgr2=0x7fa2b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2b00a2b90 0x7fa2b00a5050 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa2b802c7b0 tx=0x7fa2b8005fd0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 msgr2=0x7fa2c819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 0x7fa2c819ee20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa2c000eb10 tx=0x7fa2c000c2d0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 shutdown_connections 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fa2b00a2b90 0x7fa2b00a5050 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2c8102250 0x7fa2c819ee20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2c8101890 0x7fa2c819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 --2- 172.21.15.38:0/2452548946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2c81006a0 0x7fa2c806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.943+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 >> 172.21.15.38:0/2452548946 conn(0x7fa2c80fbc30 msgr2=0x7fa2c8104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.944+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 shutdown_connections 2024-04-22T01:16:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:35.944+0000 7fa2cd682640 1 -- 172.21.15.38:0/2452548946 wait complete. 2024-04-22T01:16:35.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T01:16:36.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 -- 172.21.15.38:0/4127429647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830103340 msgr2=0x7f4830109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 --2- 172.21.15.38:0/4127429647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830103340 0x7f4830109bd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4828009ca0 tx=0x7f482802c0f0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 -- 172.21.15.38:0/4127429647 shutdown_connections 2024-04-22T01:16:36.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 --2- 172.21.15.38:0/4127429647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830103340 0x7f4830109bd0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 --2- 172.21.15.38:0/4127429647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830102980 0x7f4830102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 --2- 172.21.15.38:0/4127429647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4830101790 0x7f4830101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 -- 172.21.15.38:0/4127429647 >> 172.21.15.38:0/4127429647 conn(0x7f48300fcd20 msgr2=0x7f48300ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 -- 172.21.15.38:0/4127429647 shutdown_connections 2024-04-22T01:16:36.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.060+0000 7f4834c20640 1 -- 172.21.15.38:0/4127429647 wait complete. 2024-04-22T01:16:36.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 Processor -- start 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 -- start start 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830101790 0x7f4830197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 0x7f4830197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4830103340 0x7f483019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f483010d410 con 0x7f4830102980 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f483010d290 con 0x7f4830101790 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f4834c20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f483010d590 con 0x7f4830103340 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 0x7f4830197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 0x7f4830197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45604/0 (socket says 172.21.15.38:45604) 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 -- 172.21.15.38:0/2558734051 learned_addr learned my addr 172.21.15.38:0/2558734051 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482ed9d640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4830103340 0x7f483019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482e59c640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830101790 0x7f4830197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 -- 172.21.15.38:0/2558734051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4830103340 msgr2=0x7f483019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4830103340 0x7f483019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 -- 172.21.15.38:0/2558734051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830101790 msgr2=0x7f4830197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830101790 0x7f4830197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482dd9b640 1 -- 172.21.15.38:0/2558734051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483019f640 con 0x7f4830102980 2024-04-22T01:16:36.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.061+0000 7f482e59c640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830101790 0x7f4830197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:36.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.062+0000 7f482dd9b640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 0x7f4830197f60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f482400e9f0 tx=0x7f482400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.067+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482400cd80 con 0x7f4830102980 2024-04-22T01:16:36.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.067+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4824010910 con 0x7f4830102980 2024-04-22T01:16:36.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.067+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f483019f930 con 0x7f4830102980 2024-04-22T01:16:36.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.067+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824018990 con 0x7f4830102980 2024-04-22T01:16:36.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.067+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f483019fed0 con 0x7f4830102980 2024-04-22T01:16:36.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.068+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4830108ab0 con 0x7f4830102980 2024-04-22T01:16:36.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.069+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4824010ac0 con 0x7f4830102980 2024-04-22T01:16:36.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.070+0000 7f481f7fe640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48180a2c10 0x7f48180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.070+0000 7f482e59c640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48180a2c10 0x7f48180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.070+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f48240cf730 con 0x7f4830102980 2024-04-22T01:16:36.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.071+0000 7f482e59c640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48180a2c10 0x7f48180a50d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f48301027e0 tx=0x7f4820005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.074+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f482409c2c0 con 0x7f4830102980 2024-04-22T01:16:36.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.259+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4830062340 con 0x7f4830102980 2024-04-22T01:16:36.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.260+0000 7f481f7fe640 1 -- 172.21.15.38:0/2558734051 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f482409bbc0 con 0x7f4830102980 2024-04-22T01:16:36.260 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48180a2c10 msgr2=0x7f48180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48180a2c10 0x7f48180a50d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f48301027e0 tx=0x7f4820005f90 comp rx=0 tx=0).stop 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 msgr2=0x7f4830197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 0x7f4830197f60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f482400e9f0 tx=0x7f482400eeb0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 shutdown_connections 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f48180a2c10 0x7f48180a50d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4830103340 0x7f483019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830102980 0x7f4830197f60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 --2- 172.21.15.38:0/2558734051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4830101790 0x7f4830197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 >> 172.21.15.38:0/2558734051 conn(0x7f48300fcd20 msgr2=0x7f4830105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 shutdown_connections 2024-04-22T01:16:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.262+0000 7f4834c20640 1 -- 172.21.15.38:0/2558734051 wait complete. 2024-04-22T01:16:36.272 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/204390032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:16:36.272 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:36 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2452548946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:16:36.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T01:16:36.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 -- 172.21.15.38:0/1589290329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c102250 msgr2=0x7eff0c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 --2- 172.21.15.38:0/1589290329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c102250 0x7eff0c108ae0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7eff04009ca0 tx=0x7eff0402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 -- 172.21.15.38:0/1589290329 shutdown_connections 2024-04-22T01:16:36.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 --2- 172.21.15.38:0/1589290329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c102250 0x7eff0c108ae0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 --2- 172.21.15.38:0/1589290329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff0c101890 0x7eff0c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 --2- 172.21.15.38:0/1589290329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c1006a0 0x7eff0c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 -- 172.21.15.38:0/1589290329 >> 172.21.15.38:0/1589290329 conn(0x7eff0c0fbc30 msgr2=0x7eff0c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 -- 172.21.15.38:0/1589290329 shutdown_connections 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.378+0000 7eff131c9640 1 -- 172.21.15.38:0/1589290329 wait complete. 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 Processor -- start 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 -- start start 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 0x7eff0c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff0c101890 0x7eff0c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c102250 0x7eff0c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10d440 con 0x7eff0c1006a0 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10d2c0 con 0x7eff0c101890 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff131c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10d5c0 con 0x7eff0c102250 2024-04-22T01:16:36.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff10f65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 0x7eff0c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff10f65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 0x7eff0c197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45626/0 (socket says 172.21.15.38:45626) 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff10f65640 1 -- 172.21.15.38:0/1131768661 learned_addr learned my addr 172.21.15.38:0/1131768661 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff0bfff640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff0c101890 0x7eff0c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff11766640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c102250 0x7eff0c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff10f65640 1 -- 172.21.15.38:0/1131768661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c102250 msgr2=0x7eff0c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff10f65640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c102250 0x7eff0c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.379+0000 7eff10f65640 1 -- 172.21.15.38:0/1131768661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff0c101890 msgr2=0x7eff0c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff10f65640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff0c101890 0x7eff0c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff10f65640 1 -- 172.21.15.38:0/1131768661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0c19f660 con 0x7eff0c1006a0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff11766640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c102250 0x7eff0c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff10f65640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 0x7eff0c197a40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7eff0000d950 tx=0x7eff0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff00017070 con 0x7eff0c1006a0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff00009c30 con 0x7eff0c1006a0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0000f600 con 0x7eff0c1006a0 2024-04-22T01:16:36.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0c19f950 con 0x7eff0c1006a0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.380+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0c19fe50 con 0x7eff0c1006a0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.381+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff0c107960 con 0x7eff0c1006a0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.382+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7eff0001a070 con 0x7eff0c1006a0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.383+0000 7eff09ffb640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efefc0a2b90 0x7efefc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.383+0000 7eff0bfff640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efefc0a2b90 0x7efefc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.383+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7eff000cef90 con 0x7eff0c1006a0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.384+0000 7eff0bfff640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efefc0a2b90 0x7efefc0a5050 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7eff0c198f50 tx=0x7efef8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.388+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7eff00009de0 con 0x7eff0c1006a0 2024-04-22T01:16:36.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/204390032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:16:36.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:36 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2452548946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:16:36.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/204390032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:16:36.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:36 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2452548946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.572+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7eff0c02d3f0 con 0x7eff0c1006a0 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.572+0000 7eff09ffb640 1 -- 172.21.15.38:0/1131768661 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7eff0009bae0 con 0x7eff0c1006a0 2024-04-22T01:16:36.572 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T01:16:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efefc0a2b90 msgr2=0x7efefc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efefc0a2b90 0x7efefc0a5050 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7eff0c198f50 tx=0x7efef8005f90 comp rx=0 tx=0).stop 2024-04-22T01:16:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 msgr2=0x7eff0c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 0x7eff0c197a40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7eff0000d950 tx=0x7eff0000de10 comp rx=0 tx=0).stop 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 shutdown_connections 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7efefc0a2b90 0x7efefc0a5050 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff0c102250 0x7eff0c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff0c101890 0x7eff0c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 --2- 172.21.15.38:0/1131768661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff0c1006a0 0x7eff0c197a40 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 >> 172.21.15.38:0/1131768661 conn(0x7eff0c0fbc30 msgr2=0x7eff0c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 shutdown_connections 2024-04-22T01:16:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.574+0000 7eff131c9640 1 -- 172.21.15.38:0/1131768661 wait complete. 2024-04-22T01:16:36.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.690+0000 7fefbd3f7640 1 -- 172.21.15.38:0/1957697575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8102980 msgr2=0x7fefb8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.690+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/1957697575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8102980 0x7fefb8102e00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009900 tx=0x7fefa002bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.690+0000 7fefbd3f7640 1 -- 172.21.15.38:0/1957697575 shutdown_connections 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.690+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/1957697575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 0x7fefb8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/1957697575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8102980 0x7fefb8102e00 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/1957697575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefb8101790 0x7fefb8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 -- 172.21.15.38:0/1957697575 >> 172.21.15.38:0/1957697575 conn(0x7fefb80fcd20 msgr2=0x7fefb80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 -- 172.21.15.38:0/1957697575 shutdown_connections 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 -- 172.21.15.38:0/1957697575 wait complete. 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 Processor -- start 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 -- start start 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 0x7fefb8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefb8102980 0x7fefb8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.691+0000 7fefbd3f7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 0x7fefb819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefbd3f7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fefb810d410 con 0x7fefb8101790 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefbd3f7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fefb810d290 con 0x7fefb8103340 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefbd3f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefb810d590 con 0x7fefb8102980 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 0x7fefb8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 0x7fefb8197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49304/0 (socket says 172.21.15.38:49304) 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 -- 172.21.15.38:0/2925424650 learned_addr learned my addr 172.21.15.38:0/2925424650 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb67fc640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefb8102980 0x7fefb8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb77fe640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 0x7fefb819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 -- 172.21.15.38:0/2925424650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefb8102980 msgr2=0x7fefb8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefb8102980 0x7fefb8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 -- 172.21.15.38:0/2925424650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 msgr2=0x7fefb819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 0x7fefb819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 -- 172.21.15.38:0/2925424650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb819f6d0 con 0x7fefb8101790 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb77fe640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 0x7fefb819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefb6ffd640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 0x7fefb8197ab0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fefb000c970 tx=0x7fefb000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0004220 con 0x7fefb8101790 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb819f9c0 con 0x7fefb8101790 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb00043c0 con 0x7fefb8101790 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0004a00 con 0x7fefb8101790 2024-04-22T01:16:36.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.692+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb819fee0 con 0x7fefb8101790 2024-04-22T01:16:36.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.693+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefb8068560 con 0x7fefb8101790 2024-04-22T01:16:36.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.694+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fefb00079b0 con 0x7fefb8101790 2024-04-22T01:16:36.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.695+0000 7fefaffff640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefa40a2b90 0x7fefa40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:36.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.695+0000 7fefb67fc640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefa40a2b90 0x7fefa40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:36.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.695+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fefb00cf2c0 con 0x7fefb8101790 2024-04-22T01:16:36.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.695+0000 7fefb67fc640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefa40a2b90 0x7fefa40a5050 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fefa0006010 tx=0x7fefa0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:36.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.700+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fefb0014030 con 0x7fefb8101790 2024-04-22T01:16:36.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.883+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fefb8062370 con 0x7fefb8101790 2024-04-22T01:16:36.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.884+0000 7fefaffff640 1 -- 172.21.15.38:0/2925424650 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fefb009be10 con 0x7fefb8101790 2024-04-22T01:16:36.884 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefa40a2b90 msgr2=0x7fefa40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefa40a2b90 0x7fefa40a5050 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fefa0006010 tx=0x7fefa0000bc0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 msgr2=0x7fefb8197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 0x7fefb8197ab0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fefb000c970 tx=0x7fefb000ce30 comp rx=0 tx=0).stop 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 shutdown_connections 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fefa40a2b90 0x7fefa40a5050 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefb8103340 0x7fefb819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefb8102980 0x7fefb8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 --2- 172.21.15.38:0/2925424650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefb8101790 0x7fefb8197ab0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:36.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 >> 172.21.15.38:0/2925424650 conn(0x7fefb80fcd20 msgr2=0x7fefb8105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:36.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 shutdown_connections 2024-04-22T01:16:36.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:36.886+0000 7fefbd3f7640 1 -- 172.21.15.38:0/2925424650 wait complete. 2024-04-22T01:16:36.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 -- 172.21.15.38:0/2901180770 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00841006a0 msgr2=0x7f0084100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 --2- 172.21.15.38:0/2901180770 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00841006a0 0x7f0084100ac0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f0074009900 tx=0x7f007402bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 -- 172.21.15.38:0/2901180770 shutdown_connections 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 --2- 172.21.15.38:0/2901180770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 0x7f0084108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 --2- 172.21.15.38:0/2901180770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0084101890 0x7f0084101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 --2- 172.21.15.38:0/2901180770 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00841006a0 0x7f0084100ac0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 -- 172.21.15.38:0/2901180770 >> 172.21.15.38:0/2901180770 conn(0x7f00840fbc10 msgr2=0x7f00840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.000+0000 7f008b06b640 1 -- 172.21.15.38:0/2901180770 shutdown_connections 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 -- 172.21.15.38:0/2901180770 wait complete. 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 Processor -- start 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 -- start start 2024-04-22T01:16:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00841006a0 0x7f0084197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 0x7f0084198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 0x7f008419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f008410d440 con 0x7f0084101890 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f008410d2c0 con 0x7f00841006a0 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f008b06b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f008410d5c0 con 0x7f0084102250 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f0083fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 0x7f0084198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f0083fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 0x7f0084198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49324/0 (socket says 172.21.15.38:49324) 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.001+0000 7f0083fff640 1 -- 172.21.15.38:0/2421942283 learned_addr learned my addr 172.21.15.38:0/2421942283 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0089608640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 0x7f008419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0088e07640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00841006a0 0x7f0084197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0083fff640 1 -- 172.21.15.38:0/2421942283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 msgr2=0x7f008419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0083fff640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 0x7f008419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0083fff640 1 -- 172.21.15.38:0/2421942283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00841006a0 msgr2=0x7f0084197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0083fff640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00841006a0 0x7f0084197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0083fff640 1 -- 172.21.15.38:0/2421942283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008419f740 con 0x7f0084101890 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0088e07640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00841006a0 0x7f0084197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0089608640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 0x7f008419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0083fff640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 0x7f0084198060 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f007800d950 tx=0x7f007800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0078017070 con 0x7f0084101890 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0078009c30 con 0x7f0084101890 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008419fa30 con 0x7f0084101890 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007800f6b0 con 0x7f0084101890 2024-04-22T01:16:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.002+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008419ff80 con 0x7f0084101890 2024-04-22T01:16:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.003+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0084107ae0 con 0x7f0084101890 2024-04-22T01:16:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.004+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f00780109d0 con 0x7f0084101890 2024-04-22T01:16:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.005+0000 7f0081ffb640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f006c0a2b90 0x7f006c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.005+0000 7f0088e07640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f006c0a2b90 0x7f006c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.005+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f00780cf0d0 con 0x7f0084101890 2024-04-22T01:16:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.006+0000 7f0088e07640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f006c0a2b90 0x7f006c0a5050 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0074036040 tx=0x7f0074000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:37.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.010+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0078009de0 con 0x7f0084101890 2024-04-22T01:16:37.154 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:37 smithi038 ceph-mon[28043]: pgmap v483: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:37.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:37 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2558734051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:37 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1131768661' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:37 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2925424650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.207+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0084062340 con 0x7f0084101890 2024-04-22T01:16:37.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.207+0000 7f0081ffb640 1 -- 172.21.15.38:0/2421942283 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+589 (secure 0 0 0) 0x7f007809bc60 con 0x7f0084101890 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3063830265 2024-04-23T01:03:15.695973+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3063830265 2024-04-23T01:03:15.695973+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2675569647 2024-04-23T01:03:15.695973+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2608977615 2024-04-23T01:03:15.695973+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3674504753 2024-04-23T01:02:47.985940+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3674504753 2024-04-23T01:02:47.985940+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/799365618 2024-04-23T01:02:47.985940+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3519840670 2024-04-23T01:02:47.985940+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/845318699 2024-04-23T01:02:47.985940+0000 2024-04-22T01:16:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/687303682 2024-04-23T01:03:15.695973+0000 2024-04-22T01:16:37.209 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2024-04-22T01:16:37.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f006c0a2b90 msgr2=0x7f006c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f006c0a2b90 0x7f006c0a5050 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0074036040 tx=0x7f0074000bc0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 msgr2=0x7f0084198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 0x7f0084198060 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f007800d950 tx=0x7f007800de10 comp rx=0 tx=0).stop 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 shutdown_connections 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f006c0a2b90 0x7f006c0a5050 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0084102250 0x7f008419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0084101890 0x7f0084198060 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 --2- 172.21.15.38:0/2421942283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00841006a0 0x7f0084197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.209+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 >> 172.21.15.38:0/2421942283 conn(0x7f00840fbc10 msgr2=0x7f0084104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.210+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 shutdown_connections 2024-04-22T01:16:37.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.210+0000 7f008b06b640 1 -- 172.21.15.38:0/2421942283 wait complete. 2024-04-22T01:16:37.220 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-22T01:16:37.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:37.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-22T01:16:37.260 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:16:37.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:37.260 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T01:16:37.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:37.316 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T01:16:37.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:37 smithi203 ceph-mon[31825]: pgmap v483: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:37.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:37 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2558734051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:37 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1131768661' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:37 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2925424650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.371 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T01:16:37.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:37.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T01:16:37.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:37 smithi111 ceph-mon[31057]: pgmap v483: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-22T01:16:37.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:37 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2558734051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:37 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1131768661' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:37 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2925424650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T01:16:37.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T01:16:37.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-04-22T01:16:37.551 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T01:16:37.562 INFO:tasks.cephadm:Teardown begin 2024-04-22T01:16:37.562 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T01:16:37.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:37.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 -- 172.21.15.38:0/595828739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 msgr2=0x7fd318102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 --2- 172.21.15.38:0/595828739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318102e20 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd300009900 tx=0x7fd30002bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 -- 172.21.15.38:0/595828739 shutdown_connections 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 --2- 172.21.15.38:0/595828739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd318103360 0x7fd318109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 --2- 172.21.15.38:0/595828739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318102e20 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 --2- 172.21.15.38:0/595828739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3181017b0 0x7fd318101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 -- 172.21.15.38:0/595828739 >> 172.21.15.38:0/595828739 conn(0x7fd3180fcd40 msgr2=0x7fd3180ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 -- 172.21.15.38:0/595828739 shutdown_connections 2024-04-22T01:16:37.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.667+0000 7fd31cb66640 1 -- 172.21.15.38:0/595828739 wait complete. 2024-04-22T01:16:37.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 Processor -- start 2024-04-22T01:16:37.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 -- start start 2024-04-22T01:16:37.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3181017b0 0x7fd318197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd318103360 0x7fd31819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810d430 con 0x7fd3181029a0 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810d2b0 con 0x7fd318103360 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31cb66640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810d5b0 con 0x7fd3181017b0 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd30ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd30ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49350/0 (socket says 172.21.15.38:49350) 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd30ffff640 1 -- 172.21.15.38:0/4281001815 learned_addr learned my addr 172.21.15.38:0/4281001815 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd31659c640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3181017b0 0x7fd318197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.668+0000 7fd316d9d640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd318103360 0x7fd31819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30ffff640 1 -- 172.21.15.38:0/4281001815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3181017b0 msgr2=0x7fd318197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30ffff640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3181017b0 0x7fd318197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30ffff640 1 -- 172.21.15.38:0/4281001815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd318103360 msgr2=0x7fd31819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30ffff640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd318103360 0x7fd31819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30ffff640 1 -- 172.21.15.38:0/4281001815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31819f630 con 0x7fd3181029a0 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd31659c640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3181017b0 0x7fd318197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:37.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30ffff640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318197f50 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd300009230 tx=0x7fd300005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:37.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300041070 con 0x7fd3181029a0 2024-04-22T01:16:37.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31819f8c0 con 0x7fd3181029a0 2024-04-22T01:16:37.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3000054e0 con 0x7fd3181029a0 2024-04-22T01:16:37.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300034440 con 0x7fd3181029a0 2024-04-22T01:16:37.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.669+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31819fe00 con 0x7fd3181029a0 2024-04-22T01:16:37.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.670+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd318108a00 con 0x7fd3181029a0 2024-04-22T01:16:37.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.671+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fd300044070 con 0x7fd3181029a0 2024-04-22T01:16:37.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.672+0000 7fd30f7fe640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3040a2b90 0x7fd3040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:37.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.672+0000 7fd31659c640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3040a2b90 0x7fd3040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:37.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.672+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7fd3000f1200 con 0x7fd3181029a0 2024-04-22T01:16:37.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.672+0000 7fd31659c640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3040a2b90 0x7fd3040a5050 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd318102800 tx=0x7fd308005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:37.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.677+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd3000bdd50 con 0x7fd3181029a0 2024-04-22T01:16:37.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.899+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd318062340 con 0x7fd3181029a0 2024-04-22T01:16:37.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.900+0000 7fd30f7fe640 1 -- 172.21.15.38:0/4281001815 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7fd3000bd650 con 0x7fd3181029a0 2024-04-22T01:16:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:16:37.900 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:16:37.901 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3040a2b90 msgr2=0x7fd3040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3040a2b90 0x7fd3040a5050 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd318102800 tx=0x7fd308005db0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 msgr2=0x7fd318197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318197f50 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd300009230 tx=0x7fd300005dd0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 shutdown_connections 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fd3040a2b90 0x7fd3040a5050 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd318103360 0x7fd31819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3181029a0 0x7fd318197f50 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 --2- 172.21.15.38:0/4281001815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3181017b0 0x7fd318197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 >> 172.21.15.38:0/4281001815 conn(0x7fd3180fcd40 msgr2=0x7fd318105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 shutdown_connections 2024-04-22T01:16:37.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:37.902+0000 7fd31cb66640 1 -- 172.21.15.38:0/4281001815 wait complete. 2024-04-22T01:16:37.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T01:16:38.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 -- 172.21.15.38:0/1178709339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c101890 msgr2=0x7f0a1c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 --2- 172.21.15.38:0/1178709339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c101890 0x7f0a1c101d10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009900 tx=0x7f0a0c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 -- 172.21.15.38:0/1178709339 shutdown_connections 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 --2- 172.21.15.38:0/1178709339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a1c102250 0x7f0a1c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 --2- 172.21.15.38:0/1178709339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c101890 0x7f0a1c101d10 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 --2- 172.21.15.38:0/1178709339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a1c1006a0 0x7f0a1c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 -- 172.21.15.38:0/1178709339 >> 172.21.15.38:0/1178709339 conn(0x7f0a1c0fbc30 msgr2=0x7f0a1c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 -- 172.21.15.38:0/1178709339 shutdown_connections 2024-04-22T01:16:38.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.017+0000 7f0a235af640 1 -- 172.21.15.38:0/1178709339 wait complete. 2024-04-22T01:16:38.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 Processor -- start 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 -- start start 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a1c1006a0 0x7f0a1c104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a1c101890 0x7f0a1c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 0x7f0a1c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c10c370 con 0x7f0a1c102250 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c10c1f0 con 0x7f0a1c1006a0 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a235af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c10c4f0 con 0x7f0a1c101890 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a21b4c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 0x7f0a1c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a2134b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a1c1006a0 0x7f0a1c104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a21b4c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 0x7f0a1c1031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49362/0 (socket says 172.21.15.38:49362) 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a20b4a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a1c101890 0x7f0a1c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.018+0000 7f0a21b4c640 1 -- 172.21.15.38:0/2820368504 learned_addr learned my addr 172.21.15.38:0/2820368504 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a21b4c640 1 -- 172.21.15.38:0/2820368504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a1c101890 msgr2=0x7f0a1c102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a21b4c640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a1c101890 0x7f0a1c102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a21b4c640 1 -- 172.21.15.38:0/2820368504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a1c1006a0 msgr2=0x7f0a1c104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a21b4c640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a1c1006a0 0x7f0a1c104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a21b4c640 1 -- 172.21.15.38:0/2820368504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c103a40 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a21b4c640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 0x7f0a1c1031b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400cad0 tx=0x7f0a1400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400ae20 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a14005020 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1c1a1bf0 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400f6f0 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.019+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1c1a2110 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.020+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a1c0689b0 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.021+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f0a140051d0 con 0x7f0a1c102250 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.022+0000 7f0a127fc640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0a080a2b90 0x7f0a080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.022+0000 7f0a2134b640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0a080a2b90 0x7f0a080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.022+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f0a140cf700 con 0x7f0a1c102250 2024-04-22T01:16:38.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.022+0000 7f0a2134b640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0a080a2b90 0x7f0a080a5050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c1016f0 tx=0x7f0a1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.026+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0a14014030 con 0x7f0a1c102250 2024-04-22T01:16:38.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.250+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a1c1a2340 con 0x7f0a1c102250 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.251+0000 7f0a127fc640 1 -- 172.21.15.38:0/2820368504 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f0a1409bc40 con 0x7f0a1c102250 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:16:38.251 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2658040728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2658040728},{"type":"v1","addr":"172.21.15.38:6835","nonce":2658040728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14532,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3419129141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3419129141},{"type":"v1","addr":"172.21.15.38:6837","nonce":3419129141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T01:10:29.943110+0000","modified":"2024-04-22T01:13:02.188765+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24485,"mds_2":24472,"mds_3":24451,"mds_4":24446},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14595":{"gid":14595,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/1149927748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1149927748},{"type":"v1","addr":"172.21.15.38:6839","nonce":1149927748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14607":{"gid":14607,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3631879100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3631879100},{"type":"v1","addr":"172.21.15.38:6841","nonce":3631879100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24400":{"gid":24400,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/3645664141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":3645664141},{"type":"v1","addr":"172.21.15.203:6833","nonce":3645664141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/1598672021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1598672021},{"type":"v1","addr":"172.21.15.111:6833","nonce":1598672021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24439":{"gid":24439,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/4027190739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4027190739},{"type":"v1","addr":"172.21.15.203:6835","nonce":4027190739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"e","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.111:6835/2761116669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2761116669},{"type":"v1","addr":"172.21.15.111:6835","nonce":2761116669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/2993816469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2993816469},{"type":"v1","addr":"172.21.15.203:6837","nonce":2993816469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.203:6839/765839756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":765839756},{"type":"v1","addr":"172.21.15.203:6839","nonce":765839756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/1636882248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":1636882248},{"type":"v1","addr":"172.21.15.111:6837","nonce":1636882248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3001874105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3001874105},{"type":"v1","addr":"172.21.15.111:6839","nonce":3001874105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24485,24527,24446,24451,24472]},"id":1}]} 2024-04-22T01:16:38.252 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0a080a2b90 msgr2=0x7f0a080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0a080a2b90 0x7f0a080a5050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c1016f0 tx=0x7f0a1800b390 comp rx=0 tx=0).stop 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 msgr2=0x7f0a1c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 0x7f0a1c1031b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400cad0 tx=0x7f0a1400cf90 comp rx=0 tx=0).stop 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 shutdown_connections 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f0a080a2b90 0x7f0a080a5050 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a1c102250 0x7f0a1c1031b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a1c101890 0x7f0a1c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 --2- 172.21.15.38:0/2820368504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a1c1006a0 0x7f0a1c104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.253+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 >> 172.21.15.38:0/2820368504 conn(0x7f0a1c0fbc30 msgr2=0x7f0a1c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.254+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 shutdown_connections 2024-04-22T01:16:38.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.254+0000 7f0a235af640 1 -- 172.21.15.38:0/2820368504 wait complete. 2024-04-22T01:16:38.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2421942283' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:16:38.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/4281001815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:16:38.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:16:38.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2421942283' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:16:38.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/4281001815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 -- 172.21.15.38:0/2916539848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978101890 msgr2=0x7f8978101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 --2- 172.21.15.38:0/2916539848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978101890 0x7f8978101d10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f896c009900 tx=0x7f896c02bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 -- 172.21.15.38:0/2916539848 shutdown_connections 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 --2- 172.21.15.38:0/2916539848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8978102250 0x7f8978108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 --2- 172.21.15.38:0/2916539848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978101890 0x7f8978101d10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 --2- 172.21.15.38:0/2916539848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 0x7f8978100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 -- 172.21.15.38:0/2916539848 >> 172.21.15.38:0/2916539848 conn(0x7f89780fbc30 msgr2=0x7f89780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 -- 172.21.15.38:0/2916539848 shutdown_connections 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.366+0000 7f897e8c2640 1 -- 172.21.15.38:0/2916539848 wait complete. 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 Processor -- start 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 -- start start 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 0x7f8978197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8978101890 0x7f8978197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 0x7f897819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f897810d440 con 0x7f8978102250 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f897810d2c0 con 0x7f89781006a0 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897e8c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f897810d5c0 con 0x7f8978101890 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 0x7f897819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 0x7f897819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49374/0 (socket says 172.21.15.38:49374) 2024-04-22T01:16:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 -- 172.21.15.38:0/496143226 learned_addr learned my addr 172.21.15.38:0/496143226 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f8977fff640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 0x7f8978197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f89777fe640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8978101890 0x7f8978197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 -- 172.21.15.38:0/496143226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8978101890 msgr2=0x7f8978197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8978101890 0x7f8978197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 -- 172.21.15.38:0/496143226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 msgr2=0x7f8978197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 0x7f8978197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f897ce5f640 1 -- 172.21.15.38:0/496143226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897819f6b0 con 0x7f8978102250 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.367+0000 7f8977fff640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 0x7f8978197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.368+0000 7f897ce5f640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 0x7f897819f040 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f897000eaa0 tx=0x7f897000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.368+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897000cdb0 con 0x7f8978102250 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.368+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8970005020 con 0x7f8978102250 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.368+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897819f9a0 con 0x7f8978102250 2024-04-22T01:16:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.368+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89700116b0 con 0x7f8978102250 2024-04-22T01:16:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.368+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897819fec0 con 0x7f8978102250 2024-04-22T01:16:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.369+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89781079c0 con 0x7f8978102250 2024-04-22T01:16:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.370+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f8970005560 con 0x7f8978102250 2024-04-22T01:16:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.371+0000 7f89757fa640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f89600a2b90 0x7f89600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.371+0000 7f8977fff640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f89600a2b90 0x7f89600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.371+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f89700cf6a0 con 0x7f8978102250 2024-04-22T01:16:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.371+0000 7f8977fff640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f89600a2b90 0x7f89600a5050 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8968005d00 tx=0x7f8968005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.375+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8970016030 con 0x7f8978102250 2024-04-22T01:16:38.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2421942283' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T01:16:38.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/4281001815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:16:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.564+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f897802d270 con 0x7f8978102250 2024-04-22T01:16:38.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.566+0000 7f89757fa640 1 -- 172.21.15.38:0/496143226 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24838 (secure 0 0 0) 0x7f897009c230 con 0x7f8978102250 2024-04-22T01:16:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T01:16:38.566 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":93,"fsid":"caf31b2e-0043-11ef-bc93-c7b262605968","created":"2024-04-22T01:02:20.325840+0000","modified":"2024-04-22T01:12:17.458713+0000","last_up_change":"2024-04-22T01:09:39.126583+0000","last_in_change":"2024-04-22T01:09:18.282166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:05:44.235660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T01:10:28.471176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T01:10:29.346433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-22T01:10:33.063023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c263cffd-6624-47e2-935b-671cc5cb79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6803","nonce":3120557304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6805","nonce":3120557304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6809","nonce":3120557304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3120557304},{"type":"v1","addr":"172.21.15.38:6807","nonce":3120557304}]},"public_addr":"172.21.15.38:6803/3120557304","cluster_addr":"172.21.15.38:6805/3120557304","heartbeat_back_addr":"172.21.15.38:6809/3120557304","heartbeat_front_addr":"172.21.15.38:6807/3120557304","state":["exists","up"]},{"osd":1,"uuid":"01d9f61a-90f2-4b76-98b8-fe7e49315113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6801","nonce":3032046140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6803","nonce":3032046140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6807","nonce":3032046140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3032046140},{"type":"v1","addr":"172.21.15.111:6805","nonce":3032046140}]},"public_addr":"172.21.15.111:6801/3032046140","cluster_addr":"172.21.15.111:6803/3032046140","heartbeat_back_addr":"172.21.15.111:6807/3032046140","heartbeat_front_addr":"172.21.15.111:6805/3032046140","state":["exists","up"]},{"osd":2,"uuid":"8ce3eb04-5c85-4391-92f6-dd636ae03c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6801","nonce":1524209627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6803","nonce":1524209627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6807","nonce":1524209627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":1524209627},{"type":"v1","addr":"172.21.15.203:6805","nonce":1524209627}]},"public_addr":"172.21.15.203:6801/1524209627","cluster_addr":"172.21.15.203:6803/1524209627","heartbeat_back_addr":"172.21.15.203:6807/1524209627","heartbeat_front_addr":"172.21.15.203:6805/1524209627","state":["exists","up"]},{"osd":3,"uuid":"e29b0ec6-0914-46b2-9887-c808bd3ab6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6811","nonce":3150547332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6813","nonce":3150547332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6817","nonce":3150547332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3150547332},{"type":"v1","addr":"172.21.15.38:6815","nonce":3150547332}]},"public_addr":"172.21.15.38:6811/3150547332","cluster_addr":"172.21.15.38:6813/3150547332","heartbeat_back_addr":"172.21.15.38:6817/3150547332","heartbeat_front_addr":"172.21.15.38:6815/3150547332","state":["exists","up"]},{"osd":4,"uuid":"05e84962-4508-4457-a9ea-c52fd31377b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6809","nonce":2127500568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6811","nonce":2127500568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6815","nonce":2127500568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2127500568},{"type":"v1","addr":"172.21.15.111:6813","nonce":2127500568}]},"public_addr":"172.21.15.111:6809/2127500568","cluster_addr":"172.21.15.111:6811/2127500568","heartbeat_back_addr":"172.21.15.111:6815/2127500568","heartbeat_front_addr":"172.21.15.111:6813/2127500568","state":["exists","up"]},{"osd":5,"uuid":"0e16f8a4-4174-4264-a7ff-976293ac88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6809","nonce":3881588829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6811","nonce":3881588829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6815","nonce":3881588829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3881588829},{"type":"v1","addr":"172.21.15.203:6813","nonce":3881588829}]},"public_addr":"172.21.15.203:6809/3881588829","cluster_addr":"172.21.15.203:6811/3881588829","heartbeat_back_addr":"172.21.15.203:6815/3881588829","heartbeat_front_addr":"172.21.15.203:6813/3881588829","state":["exists","up"]},{"osd":6,"uuid":"ed061345-8a22-457d-a3dc-13d2ee757388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6819","nonce":3905664243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6821","nonce":3905664243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6825","nonce":3905664243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3905664243},{"type":"v1","addr":"172.21.15.38:6823","nonce":3905664243}]},"public_addr":"172.21.15.38:6819/3905664243","cluster_addr":"172.21.15.38:6821/3905664243","heartbeat_back_addr":"172.21.15.38:6825/3905664243","heartbeat_front_addr":"172.21.15.38:6823/3905664243","state":["exists","up"]},{"osd":7,"uuid":"c9832e0c-b8f2-42d4-a384-a43975ec432a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6817","nonce":3257841658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6819","nonce":3257841658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6823","nonce":3257841658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3257841658},{"type":"v1","addr":"172.21.15.111:6821","nonce":3257841658}]},"public_addr":"172.21.15.111:6817/3257841658","cluster_addr":"172.21.15.111:6819/3257841658","heartbeat_back_addr":"172.21.15.111:6823/3257841658","heartbeat_front_addr":"172.21.15.111:6821/3257841658","state":["exists","up"]},{"osd":8,"uuid":"b4a5470d-0f1c-45dc-a692-cd4144caf225","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6817","nonce":3416662120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6819","nonce":3416662120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6823","nonce":3416662120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3416662120},{"type":"v1","addr":"172.21.15.203:6821","nonce":3416662120}]},"public_addr":"172.21.15.203:6817/3416662120","cluster_addr":"172.21.15.203:6819/3416662120","heartbeat_back_addr":"172.21.15.203:6823/3416662120","heartbeat_front_addr":"172.21.15.203:6821/3416662120","state":["exists","up"]},{"osd":9,"uuid":"d125ca4b-b70c-471e-864b-6eb587daf211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6827","nonce":1134107438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6829","nonce":1134107438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6833","nonce":1134107438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":1134107438},{"type":"v1","addr":"172.21.15.38:6831","nonce":1134107438}]},"public_addr":"172.21.15.38:6827/1134107438","cluster_addr":"172.21.15.38:6829/1134107438","heartbeat_back_addr":"172.21.15.38:6833/1134107438","heartbeat_front_addr":"172.21.15.38:6831/1134107438","state":["exists","up"]},{"osd":10,"uuid":"23c29567-6f8c-4bc8-b6f2-aa3e9f02ff36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6825","nonce":1666422758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6827","nonce":1666422758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6831","nonce":1666422758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1666422758},{"type":"v1","addr":"172.21.15.111:6829","nonce":1666422758}]},"public_addr":"172.21.15.111:6825/1666422758","cluster_addr":"172.21.15.111:6827/1666422758","heartbeat_back_addr":"172.21.15.111:6831/1666422758","heartbeat_front_addr":"172.21.15.111:6829/1666422758","state":["exists","up"]},{"osd":11,"uuid":"60ce13a3-6d11-48e8-b53e-609c433deaf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6825","nonce":3028852756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6827","nonce":3028852756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6831","nonce":3028852756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3028852756},{"type":"v1","addr":"172.21.15.203:6829","nonce":3028852756}]},"public_addr":"172.21.15.203:6825/3028852756","cluster_addr":"172.21.15.203:6827/3028852756","heartbeat_back_addr":"172.21.15.203:6831/3028852756","heartbeat_front_addr":"172.21.15.203:6829/3028852756","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:04:54.105075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:18.044984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:05:41.356810+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:08.038682+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:33.440940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:06:58.267491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:24.607647+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:07:50.933806+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:16.478352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:08:43.820231+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:10.271656+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T01:09:36.704266+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10},{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:6801/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3063830265":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2675569647":"2024-04-23T01:03:15.695973+0000","172.21.15.38:0/2608977615":"2024-04-23T01:03:15.695973+0000","172.21.15.38:6800/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:6801/3674504753":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/799365618":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/3519840670":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/845318699":"2024-04-23T01:02:47.985940+0000","172.21.15.38:0/687303682":"2024-04-23T01:03:15.695973+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f89600a2b90 msgr2=0x7f89600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f89600a2b90 0x7f89600a5050 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8968005d00 tx=0x7f8968005c90 comp rx=0 tx=0).stop 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 msgr2=0x7f897819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 0x7f897819f040 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f897000eaa0 tx=0x7f897000ef60 comp rx=0 tx=0).stop 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 shutdown_connections 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f89600a2b90 0x7f89600a5050 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8978102250 0x7f897819f040 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8978101890 0x7f8978197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 --2- 172.21.15.38:0/496143226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89781006a0 0x7f8978197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 >> 172.21.15.38:0/496143226 conn(0x7f89780fbc30 msgr2=0x7f8978104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 shutdown_connections 2024-04-22T01:16:38.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.568+0000 7f897e8c2640 1 -- 172.21.15.38:0/496143226 wait complete. 2024-04-22T01:16:38.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T01:16:38.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 -- 172.21.15.38:0/1488773286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c102980 msgr2=0x7f153c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 --2- 172.21.15.38:0/1488773286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c102980 0x7f153c102e00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1524009900 tx=0x7f152402bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 -- 172.21.15.38:0/1488773286 shutdown_connections 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 --2- 172.21.15.38:0/1488773286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c103340 0x7f153c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 --2- 172.21.15.38:0/1488773286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c102980 0x7f153c102e00 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 --2- 172.21.15.38:0/1488773286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c101790 0x7f153c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.682+0000 7f1541be5640 1 -- 172.21.15.38:0/1488773286 >> 172.21.15.38:0/1488773286 conn(0x7f153c0fcd20 msgr2=0x7f153c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 -- 172.21.15.38:0/1488773286 shutdown_connections 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 -- 172.21.15.38:0/1488773286 wait complete. 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 Processor -- start 2024-04-22T01:16:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 -- start start 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 0x7f153c199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c102980 0x7f153c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c103340 0x7f153c1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f153c10d3f0 con 0x7f153c101790 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.683+0000 7f1541be5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f153c10d270 con 0x7f153c102980 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f1541be5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f153c10d570 con 0x7f153c103340 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 0x7f153c199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 0x7f153c199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49400/0 (socket says 172.21.15.38:49400) 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 -- 172.21.15.38:0/1428853462 learned_addr learned my addr 172.21.15.38:0/1428853462 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153bfff640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c103340 0x7f153c1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153affd640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c102980 0x7f153c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 -- 172.21.15.38:0/1428853462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c103340 msgr2=0x7f153c1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c103340 0x7f153c1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 -- 172.21.15.38:0/1428853462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c102980 msgr2=0x7f153c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c102980 0x7f153c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 -- 172.21.15.38:0/1428853462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153c1a1930 con 0x7f153c101790 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f153b7fe640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 0x7f153c199d20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f152c00ccb0 tx=0x7f152c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c003990 con 0x7f153c101790 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153c1a1c20 con 0x7f153c101790 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f152c003b30 con 0x7f153c101790 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c00f7b0 con 0x7f153c101790 2024-04-22T01:16:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.684+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153c1a21c0 con 0x7f153c101790 2024-04-22T01:16:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.685+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f153c1a2390 con 0x7f153c101790 2024-04-22T01:16:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.687+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f152c00f950 con 0x7f153c101790 2024-04-22T01:16:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.687+0000 7f1538ff9640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15280a2b90 0x7f15280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.687+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9216+0+0 (secure 0 0 0) 0x7f152c0cf9f0 con 0x7f153c101790 2024-04-22T01:16:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.687+0000 7f153affd640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15280a2b90 0x7f15280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.688+0000 7f153affd640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15280a2b90 0x7f15280a5050 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1524006010 tx=0x7f1524000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:38.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.691+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f152c014030 con 0x7f153c101790 2024-04-22T01:16:38.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.919+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f153c1a2560 con 0x7f153c101790 2024-04-22T01:16:38.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.921+0000 7f1538ff9640 1 -- 172.21.15.38:0/1428853462 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7f152c09c580 con 0x7f153c101790 2024-04-22T01:16:38.921 INFO:teuthology.orchestra.run.smithi038.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T01:16:38.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15280a2b90 msgr2=0x7f15280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15280a2b90 0x7f15280a5050 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1524006010 tx=0x7f1524000bc0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 msgr2=0x7f153c199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 0x7f153c199d20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f152c00ccb0 tx=0x7f152c00a620 comp rx=0 tx=0).stop 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 shutdown_connections 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f15280a2b90 0x7f15280a5050 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c103340 0x7f153c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c102980 0x7f153c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 --2- 172.21.15.38:0/1428853462 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c101790 0x7f153c199d20 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 >> 172.21.15.38:0/1428853462 conn(0x7f153c0fcd20 msgr2=0x7f153c073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.923+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 shutdown_connections 2024-04-22T01:16:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:38.924+0000 7f1541be5640 1 -- 172.21.15.38:0/1428853462 wait complete. 2024-04-22T01:16:38.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.047+0000 7fce88bf7640 1 -- 172.21.15.38:0/1961275957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce841029d0 msgr2=0x7fce84102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.047+0000 7fce88bf7640 1 --2- 172.21.15.38:0/1961275957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce841029d0 0x7fce84102e50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fce78009900 tx=0x7fce7802bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 -- 172.21.15.38:0/1961275957 shutdown_connections 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 --2- 172.21.15.38:0/1961275957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce84103390 0x7fce84109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 --2- 172.21.15.38:0/1961275957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce841029d0 0x7fce84102e50 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 --2- 172.21.15.38:0/1961275957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 0x7fce84101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 -- 172.21.15.38:0/1961275957 >> 172.21.15.38:0/1961275957 conn(0x7fce840fcd30 msgr2=0x7fce840ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 -- 172.21.15.38:0/1961275957 shutdown_connections 2024-04-22T01:16:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 -- 172.21.15.38:0/1961275957 wait complete. 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 Processor -- start 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.048+0000 7fce88bf7640 1 -- start start 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 0x7fce840754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce841029d0 0x7fce84073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 0x7fce84074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fce8410d440 con 0x7fce84103390 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fce8410d2c0 con 0x7fce841029d0 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fce8410d5c0 con 0x7fce841017e0 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 0x7fce84074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 0x7fce84074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49468/0 (socket says 172.21.15.38:49468) 2024-04-22T01:16:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 -- 172.21.15.38:0/291072440 learned_addr learned my addr 172.21.15.38:0/291072440 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce81d9b640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce841029d0 0x7fce84073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce8259c640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 0x7fce840754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 -- 172.21.15.38:0/291072440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 msgr2=0x7fce840754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 0x7fce840754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 -- 172.21.15.38:0/291072440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce841029d0 msgr2=0x7fce84073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce841029d0 0x7fce84073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 -- 172.21.15.38:0/291072440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce84074900 con 0x7fce84103390 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce8259c640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 0x7fce840754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce82d9d640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 0x7fce84074070 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00eb10 tx=0x7fce7c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c004b20 con 0x7fce84103390 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce7c004cc0 con 0x7fce84103390 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce841a1b70 con 0x7fce84103390 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c010760 con 0x7fce84103390 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.049+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce841a2040 con 0x7fce84103390 2024-04-22T01:16:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.050+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce84068560 con 0x7fce84103390 2024-04-22T01:16:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.051+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7fce7c010900 con 0x7fce84103390 2024-04-22T01:16:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.052+0000 7fce737fe640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce6c0a2b90 0x7fce6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.052+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9646+0+0 (secure 0 0 0) 0x7fce7c0cfe20 con 0x7fce84103390 2024-04-22T01:16:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.052+0000 7fce8259c640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce6c0a2b90 0x7fce6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.053+0000 7fce8259c640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce6c0a2b90 0x7fce6c0a5050 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fce740058c0 tx=0x7fce74005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.055+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fce7c0536f0 con 0x7fce84103390 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l[62327]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l[62327]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mds[62331]: -- 172.21.15.203:0/793150906 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5581ba3c01a0 con 0x5581ba396800 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mds[62331]: -- 172.21.15.203:0/793150906 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581ba3c3a40 con 0x5581ba396800 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l[62327]: starting mds.l at 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f[62633]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.266 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f[62633]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mds[62637]: -- 172.21.15.203:0/1232002281 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5640148604e0 con 0x564014836800 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mds[62637]: -- 172.21.15.203:0/1232002281 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564014863a40 con 0x564014836800 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f[62633]: starting mds.f at 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: pgmap v484: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2820368504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/496143226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1428853462' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1428853462' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T01:16:39.267 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T01:16:39.268 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:39 smithi203 ceph-mon[31825]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T01:16:39.268 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c[62939]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.268 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c[62939]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.268 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mds[62943]: -- 172.21.15.203:0/855721962 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da8f1524e0 con 0x55da8f128c00 2024-04-22T01:16:39.268 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-mds[62943]: -- 172.21.15.203:0/855721962 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da8f155a40 con 0x55da8f128c00 2024-04-22T01:16:39.268 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:16:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c[62939]: starting mds.c at 2024-04-22T01:16:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.278+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fce84062340 con 0x7fce84103390 2024-04-22T01:16:39.395 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k[60675]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.395 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k[60675]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.395 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-mds[60679]: -- 172.21.15.111:0/1504261157 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563047e8da40 con 0x563047e60c00 2024-04-22T01:16:39.395 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k[60675]: starting mds.k at 2024-04-22T01:16:39.395 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h[61299]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h[61299]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h[61299]: starting mds.h at 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b[61609]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b[61609]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-mds[61613]: -- 172.21.15.111:0/2005737448 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56333fe55a40 con 0x56333fe28800 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b[61609]: starting mds.b at 2024-04-22T01:16:39.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: pgmap v484: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2820368504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/496143226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1428853462' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1428853462' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:39 smithi111 ceph-mon[31057]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e[60998]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e[60998]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-mds[61002]: -- 172.21.15.111:0/814143818 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae1b22ba40 con 0x55ae1b1fec00 2024-04-22T01:16:39.397 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:16:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e[60998]: starting mds.e at 2024-04-22T01:16:39.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a[28039]: 2024-04-22T01:16:38.919+0000 7fde2e1d7700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:16:39.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: pgmap v484: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2820368504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/496143226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1428853462' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1428853462' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:39 smithi038 ceph-mon[28043]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a[69206]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a[69206]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.405 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-mds[69210]: -- 172.21.15.38:0/761965325 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a18832fa40 con 0x55a188302c00 2024-04-22T01:16:39.406 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a[69206]: starting mds.a at 2024-04-22T01:16:39.406 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g[68847]: ignoring --setuser ceph since I am not root 2024-04-22T01:16:39.406 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g[68847]: ignoring --setgroup ceph since I am not root 2024-04-22T01:16:39.406 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-mds[68851]: -- 172.21.15.38:0/1370118420 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5604d4a65a40 con 0x5604d4a38c00 2024-04-22T01:16:39.406 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:16:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g[68847]: starting mds.g at 2024-04-22T01:16:39.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.923+0000 7fce737fe640 1 -- 172.21.15.38:0/291072440 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7fce7c09c800 con 0x7fce84103390 2024-04-22T01:16:39.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce6c0a2b90 msgr2=0x7fce6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce6c0a2b90 0x7fce6c0a5050 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fce740058c0 tx=0x7fce74005d50 comp rx=0 tx=0).stop 2024-04-22T01:16:39.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 msgr2=0x7fce84074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:39.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 0x7fce84074070 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00eb10 tx=0x7fce7c00c2d0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 shutdown_connections 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7fce6c0a2b90 0x7fce6c0a5050 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce84103390 0x7fce84074070 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce841029d0 0x7fce84073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 --2- 172.21.15.38:0/291072440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce841017e0 0x7fce840754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.926+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 >> 172.21.15.38:0/291072440 conn(0x7fce840fcd30 msgr2=0x7fce84105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.927+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 shutdown_connections 2024-04-22T01:16:39.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:39.927+0000 7fce88bf7640 1 -- 172.21.15.38:0/291072440 wait complete. 2024-04-22T01:16:39.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T01:16:40.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 -- 172.21.15.38:0/1164233497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 msgr2=0x7f4dec108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 --2- 172.21.15.38:0/1164233497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec108ac0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4de4009ca0 tx=0x7f4de402c0f0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 -- 172.21.15.38:0/1164233497 shutdown_connections 2024-04-22T01:16:40.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 --2- 172.21.15.38:0/1164233497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec108ac0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 --2- 172.21.15.38:0/1164233497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dec101870 0x7f4dec101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 --2- 172.21.15.38:0/1164233497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec100680 0x7f4dec100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 -- 172.21.15.38:0/1164233497 >> 172.21.15.38:0/1164233497 conn(0x7f4dec0fbbf0 msgr2=0x7f4dec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 -- 172.21.15.38:0/1164233497 shutdown_connections 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.059+0000 7f4df34cd640 1 -- 172.21.15.38:0/1164233497 wait complete. 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 Processor -- start 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 -- start start 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dec100680 0x7f4dec199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec101870 0x7f4dec19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10c300 con 0x7f4dec102230 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10c180 con 0x7f4dec101870 2024-04-22T01:16:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df34cd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10c480 con 0x7f4dec100680 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df1a6a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df1a6a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec1a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49476/0 (socket says 172.21.15.38:49476) 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df1a6a640 1 -- 172.21.15.38:0/2790175485 learned_addr learned my addr 172.21.15.38:0/2790175485 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df1269640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dec100680 0x7f4dec199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.060+0000 7f4df0a68640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec101870 0x7f4dec19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df1a6a640 1 -- 172.21.15.38:0/2790175485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dec100680 msgr2=0x7f4dec199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df1a6a640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dec100680 0x7f4dec199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df1a6a640 1 -- 172.21.15.38:0/2790175485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec101870 msgr2=0x7f4dec19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df1a6a640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec101870 0x7f4dec19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df1a6a640 1 -- 172.21.15.38:0/2790175485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec1a18d0 con 0x7f4dec102230 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df0a68640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec101870 0x7f4dec19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df1a6a640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec1a1230 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4de402c630 tx=0x7f4de4032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4041020 con 0x7f4dec102230 2024-04-22T01:16:40.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de4034cd0 con 0x7f4dec102230 2024-04-22T01:16:40.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dec1a1b60 con 0x7f4dec102230 2024-04-22T01:16:40.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4039040 con 0x7f4dec102230 2024-04-22T01:16:40.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.061+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dec1a2050 con 0x7f4dec102230 2024-04-22T01:16:40.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.062+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dec1a22e0 con 0x7f4dec102230 2024-04-22T01:16:40.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.063+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4de4044020 con 0x7f4dec102230 2024-04-22T01:16:40.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.064+0000 7f4de27fc640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4dd80a2b90 0x7f4dd80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:40.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.064+0000 7f4df1269640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4dd80a2b90 0x7f4dd80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:40.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.064+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9646+0+0 (secure 0 0 0) 0x7f4de40f1490 con 0x7f4dec102230 2024-04-22T01:16:40.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.065+0000 7f4df1269640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4dd80a2b90 0x7f4dd80a5050 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4dec19ab60 tx=0x7f4ddc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:40.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.069+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4de404b020 con 0x7f4dec102230 2024-04-22T01:16:40.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/291072440' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:16:40.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T01:16:40.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T01:16:40.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: Cluster is now healthy 2024-04-22T01:16:40.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/291072440' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.38:6838/3555841929,v1:172.21.15.38:6840/3555841929] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6832/789989955,v1:172.21.15.111:6834/789989955] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6833/91665833,v1:172.21.15.111:6835/91665833] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.38:6839/3503087512,v1:172.21.15.38:6841/3503087512] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6833/213429608,v1:172.21.15.203:6837/213429608] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6835/2472032534,v1:172.21.15.203:6839/2472032534] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6834/4083539703,v1:172.21.15.203:6838/4083539703] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.203:6832/2256512815,v1:172.21.15.203:6836/2256512815] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6837/1383849726,v1:172.21.15.111:6839/1383849726] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: mds.? [v2:172.21.15.111:6836/3323884149,v1:172.21.15.111:6838/3323884149] up:boot 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: fsmap 12 up:standby 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T01:16:40.236 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T01:16:40.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T01:16:40.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T01:16:40.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T01:16:40.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T01:16:40.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T01:16:40.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:40 smithi038 ceph-mon[28043]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T01:16:40.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:40.259+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4dec1a24b0 con 0x7f4dec102230 2024-04-22T01:16:40.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/291072440' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: Cluster is now healthy 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/291072440' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.38:6838/3555841929,v1:172.21.15.38:6840/3555841929] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6832/789989955,v1:172.21.15.111:6834/789989955] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6833/91665833,v1:172.21.15.111:6835/91665833] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.38:6839/3503087512,v1:172.21.15.38:6841/3503087512] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6833/213429608,v1:172.21.15.203:6837/213429608] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6835/2472032534,v1:172.21.15.203:6839/2472032534] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6834/4083539703,v1:172.21.15.203:6838/4083539703] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.203:6832/2256512815,v1:172.21.15.203:6836/2256512815] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6837/1383849726,v1:172.21.15.111:6839/1383849726] up:boot 2024-04-22T01:16:40.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: mds.? [v2:172.21.15.111:6836/3323884149,v1:172.21.15.111:6838/3323884149] up:boot 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: fsmap 12 up:standby 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T01:16:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:40 smithi111 ceph-mon[31057]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/291072440' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: Cluster is now healthy 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/291072440' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.38:6838/3555841929,v1:172.21.15.38:6840/3555841929] up:boot 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6832/789989955,v1:172.21.15.111:6834/789989955] up:boot 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6833/91665833,v1:172.21.15.111:6835/91665833] up:boot 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.38:6839/3503087512,v1:172.21.15.38:6841/3503087512] up:boot 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6833/213429608,v1:172.21.15.203:6837/213429608] up:boot 2024-04-22T01:16:40.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6835/2472032534,v1:172.21.15.203:6839/2472032534] up:boot 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6834/4083539703,v1:172.21.15.203:6838/4083539703] up:boot 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.203:6832/2256512815,v1:172.21.15.203:6836/2256512815] up:boot 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6837/1383849726,v1:172.21.15.111:6839/1383849726] up:boot 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: mds.? [v2:172.21.15.111:6836/3323884149,v1:172.21.15.111:6838/3323884149] up:boot 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: fsmap 12 up:standby 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T01:16:40.517 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T01:16:40.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:40 smithi203 ceph-mon[31825]: from='mgr.14152 172.21.15.38:0/179048906' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T01:16:41.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.088+0000 7f4de27fc640 1 -- 172.21.15.38:0/2790175485 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f4de40bdf00 con 0x7f4dec102230 2024-04-22T01:16:41.088 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' removed 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.090+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4dd80a2b90 msgr2=0x7f4dd80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4dd80a2b90 0x7f4dd80a5050 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4dec19ab60 tx=0x7f4ddc005f90 comp rx=0 tx=0).stop 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 msgr2=0x7f4dec1a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec1a1230 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4de402c630 tx=0x7f4de4032ea0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 shutdown_connections 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4dd80a2b90 0x7f4dd80a5050 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dec102230 0x7f4dec1a1230 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dec101870 0x7f4dec19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 --2- 172.21.15.38:0/2790175485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dec100680 0x7f4dec199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 >> 172.21.15.38:0/2790175485 conn(0x7f4dec0fbbf0 msgr2=0x7f4dec0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 shutdown_connections 2024-04-22T01:16:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.091+0000 7f4df34cd640 1 -- 172.21.15.38:0/2790175485 wait complete. 2024-04-22T01:16:41.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T01:16:41.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 -- 172.21.15.38:0/1254930903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888102980 msgr2=0x7f1888102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1254930903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888102980 0x7f1888102e00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1884009900 tx=0x7f188402bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:41.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 -- 172.21.15.38:0/1254930903 shutdown_connections 2024-04-22T01:16:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1254930903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1888103340 0x7f1888109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1254930903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888102980 0x7f1888102e00 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1254930903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888101790 0x7f1888101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 -- 172.21.15.38:0/1254930903 >> 172.21.15.38:0/1254930903 conn(0x7f18880fcd20 msgr2=0x7f18880ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 -- 172.21.15.38:0/1254930903 shutdown_connections 2024-04-22T01:16:41.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.213+0000 7f188f7a0640 1 -- 172.21.15.38:0/1254930903 wait complete. 2024-04-22T01:16:41.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 Processor -- start 2024-04-22T01:16:41.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 -- start start 2024-04-22T01:16:41.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 0x7f1888197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888102980 0x7f1888198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1888103340 0x7f188819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f188810d410 con 0x7f1888101790 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f188810d290 con 0x7f1888103340 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188f7a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f188810d590 con 0x7f1888102980 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188d53c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 0x7f1888197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188d53c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 0x7f1888197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49486/0 (socket says 172.21.15.38:49486) 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188d53c640 1 -- 172.21.15.38:0/1155805064 learned_addr learned my addr 172.21.15.38:0/1155805064 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.214+0000 7f188cd3b640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888102980 0x7f1888198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188d53c640 1 -- 172.21.15.38:0/1155805064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888102980 msgr2=0x7f1888198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188dd3d640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1888103340 0x7f188819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188d53c640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888102980 0x7f1888198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188d53c640 1 -- 172.21.15.38:0/1155805064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1888103340 msgr2=0x7f188819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188d53c640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1888103340 0x7f188819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188d53c640 1 -- 172.21.15.38:0/1155805064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188819f700 con 0x7f1888101790 2024-04-22T01:16:41.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188cd3b640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888102980 0x7f1888198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188d53c640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 0x7f1888197ae0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f187c00d950 tx=0x7f187c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c017070 con 0x7f1888101790 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f187c009c30 con 0x7f1888101790 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188819f9f0 con 0x7f1888101790 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c00f600 con 0x7f1888101790 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.215+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188819fef0 con 0x7f1888101790 2024-04-22T01:16:41.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.216+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1888108bd0 con 0x7f1888101790 2024-04-22T01:16:41.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.217+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f187c01a070 con 0x7f1888101790 2024-04-22T01:16:41.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.218+0000 7f187a7fc640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f18740a2b90 0x7f18740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:41.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.218+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f187c0cef30 con 0x7f1888101790 2024-04-22T01:16:41.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.218+0000 7f188cd3b640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f18740a2b90 0x7f18740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:41.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.218+0000 7f188cd3b640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f18740a2b90 0x7f18740a5050 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1884036040 tx=0x7f1884000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:41.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.224+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f187c0fd6f0 con 0x7f1888101790 2024-04-22T01:16:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:41 smithi111 ceph-mon[31057]: pgmap v486: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 409 B/s wr, 10 op/s 2024-04-22T01:16:41.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:41 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2790175485' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:41.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:41 smithi038 ceph-mon[28043]: pgmap v486: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 409 B/s wr, 10 op/s 2024-04-22T01:16:41.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:41 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2790175485' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:41.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:41.409+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f188802d170 con 0x7f1888101790 2024-04-22T01:16:41.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:41 smithi203 ceph-mon[31825]: pgmap v486: 129 pgs: 129 active+clean; 1.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 409 B/s wr, 10 op/s 2024-04-22T01:16:41.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:41 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2790175485' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:42.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.098+0000 7f187a7fc640 1 -- 172.21.15.38:0/1155805064 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f187c009de0 con 0x7f1888101790 2024-04-22T01:16:42.099 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' removed 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f18740a2b90 msgr2=0x7f18740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f18740a2b90 0x7f18740a5050 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1884036040 tx=0x7f1884000bc0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 msgr2=0x7f1888197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 0x7f1888197ae0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f187c00d950 tx=0x7f187c00de10 comp rx=0 tx=0).stop 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 shutdown_connections 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f18740a2b90 0x7f18740a5050 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1888103340 0x7f188819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1888102980 0x7f1888198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 --2- 172.21.15.38:0/1155805064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1888101790 0x7f1888197ae0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 >> 172.21.15.38:0/1155805064 conn(0x7f18880fcd20 msgr2=0x7f1888105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.102+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 shutdown_connections 2024-04-22T01:16:42.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.103+0000 7f188f7a0640 1 -- 172.21.15.38:0/1155805064 wait complete. 2024-04-22T01:16:42.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-22T01:16:42.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 -- 172.21.15.38:0/1472312165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc101880 msgr2=0x7f4bbc101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/1472312165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc101880 0x7f4bbc101d00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009900 tx=0x7f4bac02bc30 comp rx=0 tx=0).stop 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 -- 172.21.15.38:0/1472312165 shutdown_connections 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/1472312165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bbc102240 0x7f4bbc108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/1472312165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc101880 0x7f4bbc101d00 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/1472312165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 0x7f4bbc100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 -- 172.21.15.38:0/1472312165 >> 172.21.15.38:0/1472312165 conn(0x7f4bbc0fbc00 msgr2=0x7f4bbc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 -- 172.21.15.38:0/1472312165 shutdown_connections 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.231+0000 7f4bc34bb640 1 -- 172.21.15.38:0/1472312165 wait complete. 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 Processor -- start 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 -- start start 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 0x7f4bbc196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bbc101880 0x7f4bbc196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 0x7f4bbc19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc10c2f0 con 0x7f4bbc102240 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc10c170 con 0x7f4bbc101880 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc34bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc10c470 con 0x7f4bbc100690 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 0x7f4bbc19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 0x7f4bbc19dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49498/0 (socket says 172.21.15.38:49498) 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 -- 172.21.15.38:0/634111719 learned_addr learned my addr 172.21.15.38:0/634111719 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1257640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 0x7f4bbc196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc0a56640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bbc101880 0x7f4bbc196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 -- 172.21.15.38:0/634111719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 msgr2=0x7f4bbc196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 0x7f4bbc196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 -- 172.21.15.38:0/634111719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bbc101880 msgr2=0x7f4bbc196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:42.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bbc101880 0x7f4bbc196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 -- 172.21.15.38:0/634111719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bbc19e620 con 0x7f4bbc102240 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1257640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 0x7f4bbc196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.232+0000 7f4bc1a58640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 0x7f4bbc19dfb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400eaa0 tx=0x7f4bb400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.233+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400cdb0 con 0x7f4bbc102240 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.233+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb4005020 con 0x7f4bbc102240 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.233+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bbc19e910 con 0x7f4bbc102240 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.233+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40116b0 con 0x7f4bbc102240 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.233+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bbc19ee60 con 0x7f4bbc102240 2024-04-22T01:16:42.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.234+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bbc107aa0 con 0x7f4bbc102240 2024-04-22T01:16:42.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.235+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 137833+0+0 (secure 0 0 0) 0x7f4bb4005560 con 0x7f4bbc102240 2024-04-22T01:16:42.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.235+0000 7f4bb27fc640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ba80a2c10 0x7f4ba80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T01:16:42.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.236+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8475+0+0 (secure 0 0 0) 0x7f4bb40cf360 con 0x7f4bbc102240 2024-04-22T01:16:42.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.236+0000 7f4bc1257640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ba80a2c10 0x7f4ba80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T01:16:42.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.236+0000 7f4bc1257640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ba80a2c10 0x7f4ba80a50d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8005d00 tx=0x7f4bb8005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T01:16:42.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.240+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4bb4016030 con 0x7f4bbc102240 2024-04-22T01:16:42.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:42 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/2790175485' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T01:16:42.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:42 smithi111 ceph-mon[31057]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T01:16:42.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:42 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1155805064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:42.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:42 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/2790175485' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T01:16:42.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:42 smithi038 ceph-mon[28043]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T01:16:42.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:42 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1155805064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:42.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:42.427+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4bbc062340 con 0x7f4bbc102240 2024-04-22T01:16:42.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:42 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/2790175485' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T01:16:42.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:42 smithi203 ceph-mon[31825]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T01:16:42.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:42 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1155805064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:43.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.100+0000 7f4bb27fc640 1 -- 172.21.15.38:0/634111719 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f4bb409c1d0 con 0x7f4bbc102240 2024-04-22T01:16:43.100 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data_ec' removed 2024-04-22T01:16:43.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ba80a2c10 msgr2=0x7f4ba80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ba80a2c10 0x7f4ba80a50d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8005d00 tx=0x7f4bb8005c90 comp rx=0 tx=0).stop 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 msgr2=0x7f4bbc19dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 0x7f4bbc19dfb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400eaa0 tx=0x7f4bb400ef60 comp rx=0 tx=0).stop 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 shutdown_connections 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:6800/3086698001,v1:172.21.15.38:6801/3086698001] conn(0x7f4ba80a2c10 0x7f4ba80a50d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bbc102240 0x7f4bbc19dfb0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bbc101880 0x7f4bbc196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 --2- 172.21.15.38:0/634111719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bbc100690 0x7f4bbc196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 >> 172.21.15.38:0/634111719 conn(0x7f4bbc0fbc00 msgr2=0x7f4bbc104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 shutdown_connections 2024-04-22T01:16:43.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T01:16:43.102+0000 7f4bc34bb640 1 -- 172.21.15.38:0/634111719 wait complete. 2024-04-22T01:16:43.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:16:43.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:16:43.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:16:43.208 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T01:16:43.208 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T01:16:43.226 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T01:16:43.242 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T01:16:43.265 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T01:16:43.265 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T01:16:43.266 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a 2024-04-22T01:16:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:43 smithi111 ceph-mon[31057]: pgmap v488: 97 pgs: 97 active+clean; 577 KiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:16:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:43 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/1155805064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T01:16:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:43 smithi111 ceph-mon[31057]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T01:16:43.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:43 smithi111 ceph-mon[31057]: from='client.? 172.21.15.38:0/634111719' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:43.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 ceph-mon[28043]: pgmap v488: 97 pgs: 97 active+clean; 577 KiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:16:43.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/1155805064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T01:16:43.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 ceph-mon[28043]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T01:16:43.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 ceph-mon[28043]: from='client.? 172.21.15.38:0/634111719' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:43.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 systemd[1]: Stopping Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:43.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:43 smithi203 ceph-mon[31825]: pgmap v488: 97 pgs: 97 active+clean; 577 KiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T01:16:43.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:43 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/1155805064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T01:16:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:43 smithi203 ceph-mon[31825]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T01:16:43.516 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:43 smithi203 ceph-mon[31825]: from='client.? 172.21.15.38:0/634111719' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T01:16:43.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a[28039]: 2024-04-22T01:16:43.589+0000 7fde339e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:16:43.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a[28039]: 2024-04-22T01:16:43.589+0000 7fde339e2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T01:16:43.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:43 smithi038 podman[88887]: 2024-04-22 01:16:43.815195514 +0000 UTC m=+0.354224253 container died 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-22T01:16:44.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:44 smithi038 podman[88887]: 2024-04-22 01:16:44.316648956 +0000 UTC m=+0.855677750 container cleanup 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T01:16:44.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:44 smithi038 bash[88887]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a 2024-04-22T01:16:44.804 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T01:16:44.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:44 smithi038 podman[88901]: 2024-04-22 01:16:44.560188875 +0000 UTC m=+0.736768656 container remove 265dc448bbc4943d7121d3424e90b4d803328dfb39b3e309c0d07cedf3fb252a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T01:16:44.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:44 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T01:16:44.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:44 smithi038 systemd[1]: Stopped Ceph mon.a for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:16:44.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 01:16:44 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.a.service: Consumed 17.541s CPU time. 2024-04-22T01:16:44.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:44.856 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T01:16:44.856 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-22T01:16:44.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.b 2024-04-22T01:16:45.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:44 smithi111 systemd[1]: Stopping Ceph mon.b for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:45.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b[31053]: 2024-04-22T01:16:45.185+0000 7f381ac88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:16:45.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b[31053]: 2024-04-22T01:16:45.185+0000 7f381ac88700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T01:16:45.645 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:45 smithi111 podman[64643]: 2024-04-22 01:16:45.400204894 +0000 UTC m=+0.344623749 container died 1e22d75ebb5d468af3bfa99119864d4424adbdb1150ddb8f45064fb2045b17e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T01:16:46.071 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:45 smithi111 podman[64643]: 2024-04-22 01:16:45.795219248 +0000 UTC m=+0.739638113 container cleanup 1e22d75ebb5d468af3bfa99119864d4424adbdb1150ddb8f45064fb2045b17e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T01:16:46.071 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 01:16:45 smithi111 bash[64643]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-b 2024-04-22T01:16:46.221 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T01:16:46.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:46.269 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-22T01:16:46.269 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-22T01:16:46.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.c 2024-04-22T01:16:46.600 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:46 smithi203 systemd[1]: Stopping Ceph mon.c for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:47.015 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-c[31821]: 2024-04-22T01:16:46.600+0000 7f9e60f7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:16:47.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-c[31821]: 2024-04-22T01:16:46.600+0000 7f9e60f7f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T01:16:47.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:46 smithi203 podman[66329]: 2024-04-22 01:16:46.754834328 +0000 UTC m=+0.278760641 container died 11b7a1349670dd05ffc5c3df1887176bdce0c24c76d5eb40e1def1b2c777f728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T01:16:47.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:47 smithi203 podman[66329]: 2024-04-22 01:16:47.159972464 +0000 UTC m=+0.683898783 container cleanup 11b7a1349670dd05ffc5c3df1887176bdce0c24c76d5eb40e1def1b2c777f728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:16:47.410 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 01:16:47 smithi203 bash[66329]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mon-c 2024-04-22T01:16:47.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T01:16:47.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:47.623 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-22T01:16:47.623 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T01:16:47.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x 2024-04-22T01:16:47.904 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:16:47 smithi038 systemd[1]: Stopping Ceph mgr.x for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:48.280 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:16:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:16:47.972+0000 7fc4b3ac7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:16:48.281 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:16:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x[28273]: 2024-04-22T01:16:47.972+0000 7fc4b3ac7700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-22T01:16:48.281 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:16:48 smithi038 podman[89010]: 2024-04-22 01:16:48.004027241 +0000 UTC m=+0.187038082 container died 4798ca6bfa468bf17b5ac6f1c57afa91ebe89391a9ff00c4a02a941e9a6dd138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:16:48.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:16:48 smithi038 podman[89010]: 2024-04-22 01:16:48.489961504 +0000 UTC m=+0.672972332 container cleanup 4798ca6bfa468bf17b5ac6f1c57afa91ebe89391a9ff00c4a02a941e9a6dd138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-22T01:16:48.655 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 01:16:48 smithi038 bash[89010]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-x 2024-04-22T01:16:49.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T01:16:49.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:49.217 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T01:16:49.218 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T01:16:49.218 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.y 2024-04-22T01:16:49.556 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:16:49 smithi111 systemd[1]: Stopping Ceph mgr.y for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:49.837 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:16:49 smithi111 podman[64764]: 2024-04-22 01:16:49.557394222 +0000 UTC m=+0.147088726 container died 360ce42af3ecd121a36a2eedfd34d3ce8868ba5510c36bcef7f7fb5b9049faad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T01:16:50.145 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:16:49 smithi111 podman[64764]: 2024-04-22 01:16:49.920477272 +0000 UTC m=+0.510171771 container cleanup 360ce42af3ecd121a36a2eedfd34d3ce8868ba5510c36bcef7f7fb5b9049faad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T01:16:50.145 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 01:16:49 smithi111 bash[64764]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-y 2024-04-22T01:16:50.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T01:16:50.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:50.417 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T01:16:50.418 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-22T01:16:50.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.z 2024-04-22T01:16:50.743 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:16:50 smithi203 systemd[1]: Stopping Ceph mgr.z for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:51.016 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:16:50 smithi203 podman[66455]: 2024-04-22 01:16:50.744306481 +0000 UTC m=+0.130796291 container died 3277d117afd75cb0e8bfabbd1d667ab1f59c44e9a76fbf7ead5d90cef22f7b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T01:16:51.399 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:16:51 smithi203 podman[66455]: 2024-04-22 01:16:51.140887974 +0000 UTC m=+0.527377790 container cleanup 3277d117afd75cb0e8bfabbd1d667ab1f59c44e9a76fbf7ead5d90cef22f7b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8) 2024-04-22T01:16:51.399 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 01:16:51 smithi203 bash[66455]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mgr-z 2024-04-22T01:16:51.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-22T01:16:51.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:51.596 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-22T01:16:51.596 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T01:16:51.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.0 2024-04-22T01:16:51.904 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:51 smithi038 systemd[1]: Stopping Ceph osd.0 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:16:52.404 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0[42688]: 2024-04-22T01:16:51.920+0000 7f1745cb3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:16:52.404 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0[42688]: 2024-04-22T01:16:51.920+0000 7f1745cb3700 -1 osd.0 97 *** Got signal Terminated *** 2024-04-22T01:16:52.405 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0[42688]: 2024-04-22T01:16:51.920+0000 7f1745cb3700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:16:57.646 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:57 smithi038 podman[89132]: 2024-04-22 01:16:57.250869826 +0000 UTC m=+5.453785069 container died 7d0e45e7a28ec48f1c00c5528aa0b97f828b2edf6cd92916d2ed13625787a905 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:16:57.905 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:57 smithi038 podman[89132]: 2024-04-22 01:16:57.787888238 +0000 UTC m=+5.990803482 container cleanup 7d0e45e7a28ec48f1c00c5528aa0b97f828b2edf6cd92916d2ed13625787a905 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T01:16:57.905 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:57 smithi038 bash[89132]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0 2024-04-22T01:16:58.405 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:58 smithi038 podman[89150]: 2024-04-22 01:16:58.015150246 +0000 UTC m=+0.759348257 container remove 7d0e45e7a28ec48f1c00c5528aa0b97f828b2edf6cd92916d2ed13625787a905 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T01:16:58.743 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:58 smithi038 podman[89242]: 2024-04-22 01:16:58.37215105 +0000 UTC m=+0.018836604 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:16:58.743 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:58 smithi038 podman[89242]: 2024-04-22 01:16:58.508106502 +0000 UTC m=+0.154792087 container create edfd06f9858d59526106b7c19c5b125026a286a2e295138218de1dea3683f72b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T01:16:59.051 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:58 smithi038 podman[89242]: 2024-04-22 01:16:58.744443855 +0000 UTC m=+0.391129395 container init edfd06f9858d59526106b7c19c5b125026a286a2e295138218de1dea3683f72b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T01:16:59.051 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:58 smithi038 podman[89242]: 2024-04-22 01:16:58.747951404 +0000 UTC m=+0.394636947 container start edfd06f9858d59526106b7c19c5b125026a286a2e295138218de1dea3683f72b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T01:16:59.051 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:58 smithi038 podman[89242]: 2024-04-22 01:16:58.844221111 +0000 UTC m=+0.490906679 container attach edfd06f9858d59526106b7c19c5b125026a286a2e295138218de1dea3683f72b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T01:16:59.404 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:59 smithi038 podman[89242]: 2024-04-22 01:16:59.052684398 +0000 UTC m=+0.699369931 container died edfd06f9858d59526106b7c19c5b125026a286a2e295138218de1dea3683f72b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T01:16:59.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T01:16:59.861 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:59 smithi038 podman[89268]: 2024-04-22 01:16:59.808019869 +0000 UTC m=+0.749620116 container remove edfd06f9858d59526106b7c19c5b125026a286a2e295138218de1dea3683f72b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2) 2024-04-22T01:16:59.861 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:59 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-22T01:16:59.861 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:59 smithi038 systemd[1]: Stopped Ceph osd.0 for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:16:59.861 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 01:16:59 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.0.service: Consumed 31.393s CPU time. 2024-04-22T01:16:59.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:16:59.881 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T01:16:59.881 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T01:16:59.881 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.1 2024-04-22T01:17:00.191 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:16:59 smithi111 systemd[1]: Stopping Ceph osd.1 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:00.645 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1[35617]: 2024-04-22T01:17:00.190+0000 7f218dd6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:00.645 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1[35617]: 2024-04-22T01:17:00.190+0000 7f218dd6b700 -1 osd.1 97 *** Got signal Terminated *** 2024-04-22T01:17:00.645 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1[35617]: 2024-04-22T01:17:00.190+0000 7f218dd6b700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:05.896 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:05 smithi111 podman[64889]: 2024-04-22 01:17:05.603827639 +0000 UTC m=+5.531040092 container died 33adae90a82a4e1ddd642278e145b7a0aabbfc6f0e3adc2236eb109da4f128d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8) 2024-04-22T01:17:06.250 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:05 smithi111 podman[64889]: 2024-04-22 01:17:05.983808209 +0000 UTC m=+5.911020668 container cleanup 33adae90a82a4e1ddd642278e145b7a0aabbfc6f0e3adc2236eb109da4f128d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:06.251 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:05 smithi111 bash[64889]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1 2024-04-22T01:17:06.595 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:06 smithi111 podman[64920]: 2024-04-22 01:17:06.251865806 +0000 UTC m=+0.650788468 container remove 33adae90a82a4e1ddd642278e145b7a0aabbfc6f0e3adc2236eb109da4f128d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8) 2024-04-22T01:17:06.854 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:06 smithi111 podman[65009]: 2024-04-22 01:17:06.499631152 +0000 UTC m=+0.018034807 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:06.854 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:06 smithi111 podman[65009]: 2024-04-22 01:17:06.619678416 +0000 UTC m=+0.138082056 container create 17b7eeaf45c34fa83e95c5e62a84f30a29549c2b7e1e7ad6443585f4ac7152e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-22T01:17:07.145 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:06 smithi111 podman[65009]: 2024-04-22 01:17:06.981134997 +0000 UTC m=+0.499538639 container init 17b7eeaf45c34fa83e95c5e62a84f30a29549c2b7e1e7ad6443585f4ac7152e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-04-22T01:17:07.145 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:06 smithi111 podman[65009]: 2024-04-22 01:17:06.9842214 +0000 UTC m=+0.502625050 container start 17b7eeaf45c34fa83e95c5e62a84f30a29549c2b7e1e7ad6443585f4ac7152e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:07.145 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:07 smithi111 podman[65009]: 2024-04-22 01:17:07.081262552 +0000 UTC m=+0.599666201 container attach 17b7eeaf45c34fa83e95c5e62a84f30a29549c2b7e1e7ad6443585f4ac7152e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-820a2f8, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T01:17:07.576 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 01:17:07 smithi111 podman[65009]: 2024-04-22 01:17:07.271480109 +0000 UTC m=+0.789883745 container died 17b7eeaf45c34fa83e95c5e62a84f30a29549c2b7e1e7ad6443585f4ac7152e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T01:17:07.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T01:17:07.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:07.932 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T01:17:07.932 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T01:17:07.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.2 2024-04-22T01:17:08.265 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:07 smithi203 systemd[1]: Stopping Ceph osd.2 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:08.765 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2[36960]: 2024-04-22T01:17:08.269+0000 7f63d299f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:08.766 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2[36960]: 2024-04-22T01:17:08.269+0000 7f63d299f700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-22T01:17:08.766 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2[36960]: 2024-04-22T01:17:08.269+0000 7f63d299f700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:14.112 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:13 smithi203 podman[66590]: 2024-04-22 01:17:13.800313676 +0000 UTC m=+5.672727576 container died 9799ae04b533292cc8a0418111707c4085f332e2c6836e7efaec37b4162af332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:17:14.437 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:14 smithi203 podman[66590]: 2024-04-22 01:17:14.203274242 +0000 UTC m=+6.075688150 container cleanup 9799ae04b533292cc8a0418111707c4085f332e2c6836e7efaec37b4162af332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T01:17:14.437 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:14 smithi203 bash[66590]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2 2024-04-22T01:17:14.764 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:14 smithi203 podman[66608]: 2024-04-22 01:17:14.438226503 +0000 UTC m=+0.635047301 container remove 9799ae04b533292cc8a0418111707c4085f332e2c6836e7efaec37b4162af332 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:17:15.016 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:14 smithi203 podman[66699]: 2024-04-22 01:17:14.668011575 +0000 UTC m=+0.017704175 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:15.016 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:14 smithi203 podman[66699]: 2024-04-22 01:17:14.805820581 +0000 UTC m=+0.155513170 container create 7d9b0e738b0b296b6fbfa0eb1baf943d8152527d1b7b2bd0dc352461567aab1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:17:15.336 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:15 smithi203 podman[66699]: 2024-04-22 01:17:15.042647032 +0000 UTC m=+0.392339632 container init 7d9b0e738b0b296b6fbfa0eb1baf943d8152527d1b7b2bd0dc352461567aab1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:17:15.336 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:15 smithi203 podman[66699]: 2024-04-22 01:17:15.046060937 +0000 UTC m=+0.395753529 container start 7d9b0e738b0b296b6fbfa0eb1baf943d8152527d1b7b2bd0dc352461567aab1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T01:17:15.336 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:15 smithi203 podman[66699]: 2024-04-22 01:17:15.142320249 +0000 UTC m=+0.492012848 container attach 7d9b0e738b0b296b6fbfa0eb1baf943d8152527d1b7b2bd0dc352461567aab1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T01:17:15.765 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 01:17:15 smithi203 podman[66699]: 2024-04-22 01:17:15.337092653 +0000 UTC m=+0.686785246 container died 7d9b0e738b0b296b6fbfa0eb1baf943d8152527d1b7b2bd0dc352461567aab1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-22T01:17:16.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T01:17:16.181 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:16.181 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T01:17:16.181 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T01:17:16.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.3 2024-04-22T01:17:16.502 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:16 smithi038 systemd[1]: Stopping Ceph osd.3 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:16.827 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:16 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:17:16.501+0000 7fbe040f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:16.827 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:16 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:17:16.501+0000 7fbe040f1700 -1 osd.3 97 *** Got signal Terminated *** 2024-04-22T01:17:16.828 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:16 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:17:16.501+0000 7fbe040f1700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:18.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:18 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:18.556+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:18.904 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:18 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:17:18.648+0000 7fbdfc6dd700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.026455+0000 front 2024-04-22T01:16:56.026540+0000 (oldest deadline 2024-04-22T01:17:17.726629+0000) 2024-04-22T01:17:19.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:19 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:19.546+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:19.904 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:19 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:17:19.623+0000 7fbdfc6dd700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.026455+0000 front 2024-04-22T01:16:56.026540+0000 (oldest deadline 2024-04-22T01:17:17.726629+0000) 2024-04-22T01:17:20.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:20 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:20.229+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:20.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:20 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:20.516+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:20.904 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:20 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3[49340]: 2024-04-22T01:17:20.585+0000 7fbdfc6dd700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.026455+0000 front 2024-04-22T01:16:56.026540+0000 (oldest deadline 2024-04-22T01:17:17.726629+0000) 2024-04-22T01:17:21.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:21.093+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:21.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:21 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:21.201+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:21.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:21 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:21.466+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:22.128 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:21 smithi038 podman[89348]: 2024-04-22 01:17:21.861006087 +0000 UTC m=+5.479474244 container died 6bb333755e2fd614084f503e8e2501245f7df5827c384218f1a418720c83cfe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-22T01:17:22.405 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:22 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:22.168+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:22.405 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:22 smithi038 podman[89348]: 2024-04-22 01:17:22.319848472 +0000 UTC m=+5.938316675 container cleanup 6bb333755e2fd614084f503e8e2501245f7df5827c384218f1a418720c83cfe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T01:17:22.405 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:22 smithi038 bash[89348]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3 2024-04-22T01:17:22.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:22.133+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:22.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:22.107+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:22.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:22 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:22.495+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:22.904 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:22 smithi038 podman[89376]: 2024-04-22 01:17:22.607068277 +0000 UTC m=+0.740791399 container remove 6bb333755e2fd614084f503e8e2501245f7df5827c384218f1a418720c83cfe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:17:23.218 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:23 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:23.124+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:23.218 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:22 smithi038 podman[89465]: 2024-04-22 01:17:22.860723694 +0000 UTC m=+0.017837705 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:23.218 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:22 smithi038 podman[89465]: 2024-04-22 01:17:22.974668639 +0000 UTC m=+0.131782677 container create a463b1c40395a74c1ad6bc459eca460c072e549b31c19eb2d3f1ca82ead88789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T01:17:23.218 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:23 smithi038 podman[89465]: 2024-04-22 01:17:23.21927596 +0000 UTC m=+0.376389964 container init a463b1c40395a74c1ad6bc459eca460c072e549b31c19eb2d3f1ca82ead88789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T01:17:23.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:23.165+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:23.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:23.146+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:23.527 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:23 smithi038 podman[89465]: 2024-04-22 01:17:23.223162182 +0000 UTC m=+0.380276181 container start a463b1c40395a74c1ad6bc459eca460c072e549b31c19eb2d3f1ca82ead88789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:17:23.527 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:23 smithi038 podman[89465]: 2024-04-22 01:17:23.302261103 +0000 UTC m=+0.459375111 container attach a463b1c40395a74c1ad6bc459eca460c072e549b31c19eb2d3f1ca82ead88789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T01:17:23.837 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 01:17:23 smithi038 podman[89465]: 2024-04-22 01:17:23.528684892 +0000 UTC m=+0.685798889 container died a463b1c40395a74c1ad6bc459eca460c072e549b31c19eb2d3f1ca82ead88789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:17:23.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:23 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:23.508+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:24.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T01:17:24.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:24.192 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T01:17:24.193 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T01:17:24.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.4 2024-04-22T01:17:24.404 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:24 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:24.167+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:24.504 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:24 smithi111 systemd[1]: Stopping Ceph osd.4 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:24.515 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:24.147+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:24.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:24.132+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:24.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:24 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:24.507+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:24.895 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:24 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:17:24.504+0000 7f2ba6715700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:24.895 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:24 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:17:24.504+0000 7f2ba6715700 -1 osd.4 97 *** Got signal Terminated *** 2024-04-22T01:17:24.895 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:24 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:17:24.504+0000 7f2ba6715700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:25.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:25.172+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:25.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:25.116+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:25.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:25 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:25.199+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:25.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:25 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:25.556+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:26.515 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:26 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:26.174+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:26.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:26 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:26.084+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:26.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:26 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:26.084+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:26.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:26 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:26.198+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:26.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:26 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:26.586+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:27.168 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:27.046+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:27.168 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:27.046+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:27.404 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:27 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:27.213+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:27.478 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:27.168+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:27.616 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:27 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:17:27.347+0000 7f2ba2d09700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.434084+0000 front 2024-04-22T01:17:04.433930+0000 (oldest deadline 2024-04-22T01:17:27.334111+0000) 2024-04-22T01:17:27.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:27 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:27.478+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:27.898 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:27 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:27.778+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:27.898 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:27 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:27.616+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:27.899 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:27 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:27.616+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:28.458 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:28 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:28.217+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:28.458 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:28 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:28.077+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:28.458 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:28 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:28.077+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:28.616 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:28 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:17:28.356+0000 7f2ba2d09700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.434084+0000 front 2024-04-22T01:17:04.433930+0000 (oldest deadline 2024-04-22T01:17:27.334111+0000) 2024-04-22T01:17:28.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:28 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:28.218+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:28.727 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:28 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:28.458+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:28.895 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:28 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:28.776+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:28.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:28 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:28.616+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:28.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:28 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:28.616+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:29.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:28 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:28.748+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:29.482 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:29.170+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:29.482 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:29.072+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:29.482 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:29.072+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:29.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:29 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4[42114]: 2024-04-22T01:17:29.348+0000 7f2ba2d09700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.434084+0000 front 2024-04-22T01:17:04.433930+0000 (oldest deadline 2024-04-22T01:17:27.334111+0000) 2024-04-22T01:17:29.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:29 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:29.266+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:29.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:29 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:29.482+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:30.032 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:29 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:29.810+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:30.032 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:29 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:29.649+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:30.032 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:29 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:29.649+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:30.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:29 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:29.720+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:30.283 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:30 smithi111 podman[65116]: 2024-04-22 01:17:30.032778674 +0000 UTC m=+5.649951798 container died 650a73fe5049be0e867d88fc0139b877dec9fb7bb02ff68622cb65807ecf05e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:17:30.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:30.190+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:30.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:30.190+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:30.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:30.045+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:30.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:30.045+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:30.547 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:30 smithi111 podman[65116]: 2024-04-22 01:17:30.446708121 +0000 UTC m=+6.063881243 container cleanup 650a73fe5049be0e867d88fc0139b877dec9fb7bb02ff68622cb65807ecf05e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-22T01:17:30.547 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:30 smithi111 bash[65116]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4 2024-04-22T01:17:30.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:30.284+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:30.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:30.284+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:30.895 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:30 smithi111 podman[65136]: 2024-04-22 01:17:30.71504723 +0000 UTC m=+0.682474427 container remove 650a73fe5049be0e867d88fc0139b877dec9fb7bb02ff68622cb65807ecf05e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T01:17:30.896 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:30 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:30.622+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:30.896 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:30 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:30.622+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:30.896 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:30 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:30.778+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:30.999 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:30 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:30.524+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:31.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:30 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:30.706+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:31.265 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:31.206+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:31.266 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:31.206+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:31.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:30.999+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:31.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:30.999+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:31.368 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:31 smithi111 podman[65233]: 2024-04-22 01:17:30.961569516 +0000 UTC m=+0.018000384 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:31.368 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:31 smithi111 podman[65233]: 2024-04-22 01:17:31.0994613 +0000 UTC m=+0.155892155 container create e0798fe7231df2dab6dd1290d251334237774df752a10d34040b7b74e54363eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:17:31.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:31 smithi111 podman[65233]: 2024-04-22 01:17:31.369107329 +0000 UTC m=+0.425538190 container init e0798fe7231df2dab6dd1290d251334237774df752a10d34040b7b74e54363eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:31.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:31 smithi111 podman[65233]: 2024-04-22 01:17:31.37258657 +0000 UTC m=+0.429017429 container start e0798fe7231df2dab6dd1290d251334237774df752a10d34040b7b74e54363eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:31.645 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:31 smithi111 podman[65233]: 2024-04-22 01:17:31.469061919 +0000 UTC m=+0.525492776 container attach e0798fe7231df2dab6dd1290d251334237774df752a10d34040b7b74e54363eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:17:31.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:31 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:31.285+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:31.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:31 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:31.285+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:32.016 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:31 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:31.558+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:32.021 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:31 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:31.810+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:32.021 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:31 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:31.670+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:32.021 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:31 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:31.670+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:32.021 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 01:17:31 smithi111 podman[65233]: 2024-04-22 01:17:31.668765776 +0000 UTC m=+0.725196627 container died e0798fe7231df2dab6dd1290d251334237774df752a10d34040b7b74e54363eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:32.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:31 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:31.690+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:32.365 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T01:17:32.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:32.405 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T01:17:32.405 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T01:17:32.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.5 2024-04-22T01:17:32.433 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:32.249+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:32.433 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:32.249+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:32.433 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:32.046+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:32.433 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:32.046+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:32.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:32.245+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:32.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:32.245+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:32.723 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:32.530+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:32.723 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:32 smithi203 systemd[1]: Stopping Ceph osd.5 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:32.723 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:32.723+0000 7fdac9547700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:33.015 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:32.723+0000 7fdac9547700 -1 osd.5 97 *** Got signal Terminated *** 2024-04-22T01:17:33.015 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:32 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:32.723+0000 7fdac9547700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:33.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:32.825+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:33.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:32.668+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:33.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:32 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:32.668+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:33.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:32.707+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:33.515 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:33.037+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:33.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:33.037+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:33.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:33.264+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:33.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:33.264+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:33.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:33.228+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:33.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:33.228+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:34.006 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:33 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:33.554+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:34.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:33 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:33.777+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:34.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:33 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:33.648+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:34.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:33 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:33.648+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:34.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:33 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:33.698+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:34.265 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:34.217+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:34.266 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:34.217+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:34.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:34.007+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:34.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:34.007+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:34.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:34 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:34.181+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:34.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:34 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:34.181+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:34.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:34.508+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:34.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:34 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:34.508+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:35.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:34.760+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:35.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:34.662+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:35.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:34 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:34.662+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:35.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:34 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:34.699+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:35.515 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:35 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:35.265+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:35.516 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:35 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:35.265+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:35.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:35 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:35.043+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:35.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:35 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:35.043+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:35.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:35 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:35.198+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:35.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:35 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:35.198+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:35.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:35 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:35.198+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:36.013 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:35 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:35.517+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:36.013 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:35 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:35.517+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:36.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:35.719+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:36.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:35.694+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:36.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:35.694+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:36.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:35 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:35.694+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:36.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:35 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:35.659+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:36.265 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:36.226+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:36.266 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:36.226+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:36.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:36.014+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:36.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:36.014+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:36.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:36.014+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:36.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:36 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:36.208+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:36.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:36 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:36.208+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:36.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:36 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:36.208+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:36.980 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:36.518+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:36.981 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:36.518+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:37.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:36.722+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:37.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:36.722+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:37.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:36.697+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:37.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:36.697+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:37.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:36 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:36.697+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:37.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:36 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:36.676+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:37.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:36.981+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:37.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:36.981+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:37.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:36 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:36.981+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:37.644 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:37 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:37.159+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:37.644 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:37 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:37.159+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:37.644 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:37 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:37.159+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:37.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:37.489+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:37.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:37.489+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:37.766 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:37.274+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.470119+0000 front 2024-04-22T01:16:56.470204+0000 (oldest deadline 2024-04-22T01:17:20.570299+0000) 2024-04-22T01:17:37.766 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5[43745]: 2024-04-22T01:17:37.274+0000 7fdac5b3b700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:00.570872+0000 front 2024-04-22T01:17:00.570923+0000 (oldest deadline 2024-04-22T01:17:29.371410+0000) 2024-04-22T01:17:37.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:37 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:37.644+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:38.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:37 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:37.698+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:38.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:37 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:37.698+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:38.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:37 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:37.694+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:38.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:37 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:37.694+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:38.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:37 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:37.694+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:38.265 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:38 smithi203 podman[66828]: 2024-04-22 01:17:38.081043582 +0000 UTC m=+5.476912580 container died 4aebeda6765e863c0a2228609ac0b427f25d253c775da0640946d3e620623a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T01:17:38.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:37.957+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:38.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:37.957+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:38.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:37 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:37.957+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:38.621 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:38.192+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:38.621 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:38.192+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:38.621 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:38.192+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:38.709 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:38.443+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:38.709 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:38.443+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:38.710 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:38 smithi203 podman[66828]: 2024-04-22 01:17:38.458835547 +0000 UTC m=+5.854704537 container cleanup 4aebeda6765e863c0a2228609ac0b427f25d253c775da0640946d3e620623a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T01:17:38.710 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:38 smithi203 bash[66828]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5 2024-04-22T01:17:38.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:38.621+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:38.905 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:38 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:38.621+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:39.015 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:38 smithi203 podman[66845]: 2024-04-22 01:17:38.710789609 +0000 UTC m=+0.628573236 container remove 4aebeda6765e863c0a2228609ac0b427f25d253c775da0640946d3e620623a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T01:17:39.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:38.915+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:39.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:38.915+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:39.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:38 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:38.915+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:39.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:38.691+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:39.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:38.691+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:39.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:38.665+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:39.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:38.665+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:39.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:38 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:38.665+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:39.299 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:39 smithi203 podman[66941]: 2024-04-22 01:17:38.933864586 +0000 UTC m=+0.018279941 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:39.299 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:39 smithi203 podman[66941]: 2024-04-22 01:17:39.072540084 +0000 UTC m=+0.156955418 container create 4ed65183796c94ecffea03aa41a2b77da30f2ba39160ef36934784fd59fc0702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T01:17:39.299 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:39 smithi203 podman[66941]: 2024-04-22 01:17:39.300381835 +0000 UTC m=+0.384797168 container init 4ed65183796c94ecffea03aa41a2b77da30f2ba39160ef36934784fd59fc0702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T01:17:39.602 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:39 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:39.399+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:39.602 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:39 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:39.399+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:39.603 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:39 smithi203 podman[66941]: 2024-04-22 01:17:39.303518015 +0000 UTC m=+0.387933343 container start 4ed65183796c94ecffea03aa41a2b77da30f2ba39160ef36934784fd59fc0702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:39.603 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:39 smithi203 podman[66941]: 2024-04-22 01:17:39.408765001 +0000 UTC m=+0.493180342 container attach 4ed65183796c94ecffea03aa41a2b77da30f2ba39160ef36934784fd59fc0702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-22T01:17:39.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:39 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:39.237+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:39.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:39 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:39.237+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:39.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:39 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:39.237+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:39.902 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 01:17:39 smithi203 podman[66941]: 2024-04-22 01:17:39.604203866 +0000 UTC m=+0.688619205 container died 4ed65183796c94ecffea03aa41a2b77da30f2ba39160ef36934784fd59fc0702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T01:17:40.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:39 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:39.647+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:40.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:39 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:39.647+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:40.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:39 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:39.686+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:40.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:39 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:39.686+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:40.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:39 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:39.686+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:40.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:39 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:39.660+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:40.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:39 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:39.660+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:40.203 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:39 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:39.914+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:40.204 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:39 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:39.914+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:40.204 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:39 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:39.914+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:40.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T01:17:40.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:40.296 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T01:17:40.296 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T01:17:40.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.6 2024-04-22T01:17:40.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:40 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:40.443+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:40.516 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:40 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:40.443+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:40.608 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:40.287+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:40.608 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:40.287+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:40.608 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:40.287+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:40.609 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:40 smithi038 systemd[1]: Stopping Ceph osd.6 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:40.609 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:40.608+0000 7f98e39e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:40.609 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:40.608+0000 7f98e39e6700 -1 osd.6 97 *** Got signal Terminated *** 2024-04-22T01:17:40.609 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:40.608+0000 7f98e39e6700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:40.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:40.619+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:40.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:40.619+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:41.035 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:40 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:40.696+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:41.035 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:40 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:40.696+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:41.035 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:40 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:40.726+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:41.035 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:40 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:40.726+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:41.035 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:40 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:40.726+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:41.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:40 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:40.961+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:41.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:40 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:40.961+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:41.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:40 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:40.961+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:41.620 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:41.240+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:41.620 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:41.240+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:41.620 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:41.240+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:41.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:41 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:41.393+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:41.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:41 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:41.393+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:41.895 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:41 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:41.736+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:41.895 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:41 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:41.736+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:41.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:41 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:41.704+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:41.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:41 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:41.704+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:41.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:41 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:41.704+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:41.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:41.620+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:41.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:41 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:41.620+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:42.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:41 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:41.936+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:42.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:41 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:41.936+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:42.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:41 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:41.936+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:42.643 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:42 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:42.250+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:42.643 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:42 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:42.250+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:42.643 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:42 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:42.250+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:42.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:42.435+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:42.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:42.435+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:42.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:42 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:42.644+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:42.905 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:42 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:42.644+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:43.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:42 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:42.699+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:43.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:42 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:42.699+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:43.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:42 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:42.674+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:43.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:42 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:42.674+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:43.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:42 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:42.674+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:43.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:42.960+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:43.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:42.960+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:43.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:42 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:42.960+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:43.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:43.299+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:43.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:43.299+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:43.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:43.299+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:43.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:43 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:43.471+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:43.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:43 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:43.471+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:43.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:43 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:43.471+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:44.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:43 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:43.695+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:44.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:43 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:43.695+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:44.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:43 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:43.680+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:44.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:43 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:43.680+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:44.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:43 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:43.680+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:44.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:43.690+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:44.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:43 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:43.690+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:44.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:43 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:43.956+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:44.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:43 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:43.956+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:44.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:43 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:43.956+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:44.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:44.343+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:44.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:44.343+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:44.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:44.343+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:44.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:44.343+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:44.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:44 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:44.449+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:44.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:44 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:44.449+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:44.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:44 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:44.449+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:45.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:44 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:44.726+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:45.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:44 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:44.726+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:45.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:44 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:44.726+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:45.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:44 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:44.646+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:45.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:44 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:44.646+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:45.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:44 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:44.646+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:45.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:44.689+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.149643+0000 front 2024-04-22T01:17:02.149519+0000 (oldest deadline 2024-04-22T01:17:28.049595+0000) 2024-04-22T01:17:45.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:44.689+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.552556+0000 front 2024-04-22T01:17:12.552218+0000 (oldest deadline 2024-04-22T01:17:38.452246+0000) 2024-04-22T01:17:45.154 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:44 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6[55883]: 2024-04-22T01:17:44.689+0000 7f98dbfd2700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:18.452585+0000 front 2024-04-22T01:17:18.452660+0000 (oldest deadline 2024-04-22T01:17:43.752788+0000) 2024-04-22T01:17:45.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:44 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:44.960+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:45.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:44 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:44.960+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:45.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:44 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:44.960+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:45.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:45.334+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:45.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:45.334+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:45.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:45.334+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:45.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:45 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:45.334+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:45.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:45 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:45.465+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:45.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:45 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:45.465+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:45.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:45 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:45.465+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:46.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:45.730+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:46.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:45.730+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:46.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:45.730+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:46.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:45.684+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:46.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:45.684+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:46.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:45.684+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:46.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:45.684+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:46.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:45 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:45.977+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:46.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:45 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:45.977+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:46.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:45 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:45.977+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:46.302 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:45 smithi038 podman[89571]: 2024-04-22 01:17:45.938021059 +0000 UTC m=+5.440674290 container died f759c73fac7579f7331cb6e2949fca8adcd716c235acf45494f19b6494d8d0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T01:17:46.583 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:46.303+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:46.583 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:46.303+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:46.583 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:46.303+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:46.583 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:46 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:46.303+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:46.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:46.455+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:46.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:46.455+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:46.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:46.455+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:46.904 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:46 smithi038 podman[89571]: 2024-04-22 01:17:46.584433912 +0000 UTC m=+6.087087145 container cleanup f759c73fac7579f7331cb6e2949fca8adcd716c235acf45494f19b6494d8d0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T01:17:46.905 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:46 smithi038 bash[89571]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6 2024-04-22T01:17:46.905 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:46 smithi038 podman[89588]: 2024-04-22 01:17:46.777707469 +0000 UTC m=+0.833709788 container remove f759c73fac7579f7331cb6e2949fca8adcd716c235acf45494f19b6494d8d0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:17:47.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:46.666+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:47.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:46.666+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:47.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:46.666+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:47.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:46.666+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:47.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:46.754+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:47.146 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:46.754+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:47.146 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:46 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:46.754+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:47.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:46.965+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:47.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:46.965+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:47.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:46 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:46.965+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:47.513 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:47.349+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:47.513 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:47.349+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:47.513 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:47.349+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:47.513 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:47 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:47.349+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:47.514 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:47 smithi038 podman[89674]: 2024-04-22 01:17:47.119069517 +0000 UTC m=+0.018518173 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:47.514 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:47 smithi038 podman[89674]: 2024-04-22 01:17:47.22056132 +0000 UTC m=+0.120009957 container create cb1d5f4a538df68c0712e7b0619c5c55f4399d72de2aae085c03047ea6637710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T01:17:47.514 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:47 smithi038 podman[89674]: 2024-04-22 01:17:47.423048635 +0000 UTC m=+0.322497273 container init cb1d5f4a538df68c0712e7b0619c5c55f4399d72de2aae085c03047ea6637710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T01:17:47.514 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:47 smithi038 podman[89674]: 2024-04-22 01:17:47.426355085 +0000 UTC m=+0.325803723 container start cb1d5f4a538df68c0712e7b0619c5c55f4399d72de2aae085c03047ea6637710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:47.514 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:47 smithi038 podman[89674]: 2024-04-22 01:17:47.514679645 +0000 UTC m=+0.414128295 container attach cb1d5f4a538df68c0712e7b0619c5c55f4399d72de2aae085c03047ea6637710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:17:47.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:47 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:47.406+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:47.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:47 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:47.406+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:47.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:47 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:47.406+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:47.905 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 01:17:47 smithi038 podman[89674]: 2024-04-22 01:17:47.729278921 +0000 UTC m=+0.628727566 container died cb1d5f4a538df68c0712e7b0619c5c55f4399d72de2aae085c03047ea6637710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:47.772+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:47.772+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:47.772+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:47.675+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:47.675+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:47.675+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:48.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:47 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:47.675+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:48.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:47 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:47.946+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:48.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:47 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:47.946+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:48.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:47 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:47.946+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:48.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T01:17:48.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:48.448 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T01:17:48.448 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T01:17:48.448 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.7 2024-04-22T01:17:48.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:48 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:48.394+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:48.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:48 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:48.394+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:48.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:48 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:48.394+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:48.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:48 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:48.394+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:48.741 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 systemd[1]: Stopping Ceph osd.7 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:48.741 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:48.742+0000 7ffb0a7d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:48.741 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:48.742+0000 7ffb0a7d8700 -1 osd.7 97 *** Got signal Terminated *** 2024-04-22T01:17:48.742 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:48.742+0000 7ffb0a7d8700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:48.742 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:48.711+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:48.742 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:48.711+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:48.742 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:48.711+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:48.742 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:48.711+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:48.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:48 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:48.425+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:48.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:48 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:48.426+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:48.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:48 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:48.426+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:49.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:48.811+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:49.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:48.811+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:49.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:48.811+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:49.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:48 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:48.957+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:49.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:48 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:48.957+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:49.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:48 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:48.957+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:49.650 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:49 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:49.373+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:49.650 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:49 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:49.373+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:49.650 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:49 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:49.373+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:49.650 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:49 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:49.373+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:49.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:49 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:49.385+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:49.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:49 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:49.385+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:49.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:49 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:49.385+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:50.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:49.782+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:50.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:49.782+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:50.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:49.782+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:50.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:49.678+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:50.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:49.678+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:50.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:49.678+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:50.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:49 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:49.678+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:50.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:49 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:49.927+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:50.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:49 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:49.927+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:50.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:49 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:49.927+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:50.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:50 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:50.399+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:50.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:50 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:50.399+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:50.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:50 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:50.399+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:50.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:50 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:50.399+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:50.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:50 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:50.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:50.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:50 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:50.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:50.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:50 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:50.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:50.810 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:50.710+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:50.810 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:50.710+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:50.810 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:50.710+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:50.811 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:50.710+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:50.811 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:50.809+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:50.811 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:50.809+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:50.811 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:50.809+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:50.811 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:50 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:50.809+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.343939+0000 front 2024-04-22T01:17:27.343980+0000 (oldest deadline 2024-04-22T01:17:50.244196+0000) 2024-04-22T01:17:51.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:50 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:50.883+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:51.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:50 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:50.883+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:51.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:50 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:50.883+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:51.654 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:51.369+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:51.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:51.369+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:51.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:51.369+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:51.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:51.369+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:51.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:51 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:51.369+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:51.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:51.357+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:51.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:51.357+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:51.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:51.357+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:51.845+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:51.845+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:51.845+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:51.845+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.343939+0000 front 2024-04-22T01:17:27.343980+0000 (oldest deadline 2024-04-22T01:17:50.244196+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:51.754+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:51.754+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:52.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:51.754+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:52.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:51 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:51.754+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:52.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:51.846+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:52.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:51.846+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:52.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:51.846+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:52.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:51 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:51.846+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:52.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:52.393+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:52.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:52.393+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:52.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:52.393+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:52.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:52.393+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:52.655 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:52 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:52.393+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:52.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:52.352+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:52.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:52.352+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:52.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:52.352+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:53.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:52.842+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:01.138981+0000 front 2024-04-22T01:17:01.139004+0000 (oldest deadline 2024-04-22T01:17:27.539844+0000) 2024-04-22T01:17:53.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:52.842+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:12.841101+0000 front 2024-04-22T01:17:12.841141+0000 (oldest deadline 2024-04-22T01:17:35.740954+0000) 2024-04-22T01:17:53.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:52.842+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.342945+0000 front 2024-04-22T01:17:20.343080+0000 (oldest deadline 2024-04-22T01:17:43.842992+0000) 2024-04-22T01:17:53.145 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7[48907]: 2024-04-22T01:17:52.842+0000 7ffb06dcc700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.343939+0000 front 2024-04-22T01:17:27.343980+0000 (oldest deadline 2024-04-22T01:17:50.244196+0000) 2024-04-22T01:17:53.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:52.752+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:53.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:52.752+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:53.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:52.752+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:53.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:52.752+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:53.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:52.866+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:53.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:52.866+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:53.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:52.866+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:53.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:52 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:52.866+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:53.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:53.387+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:53.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:53.387+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:53.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:53.387+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:53.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:53.387+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:53.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:53 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:53.434+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:53.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:53 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:53.434+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:53.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:53 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:53.434+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:53.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:53 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:53.434+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:53.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:53 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:53.434+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:54.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:53 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:53.773+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:54.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:53 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:53.773+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:54.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:53 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:53.773+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:54.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:53 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:53.773+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:54.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:53 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:53.773+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:17:54.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:53.820+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:54.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:53.820+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:54.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:53.820+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:54.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:53 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:53.820+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:54.592 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:54 smithi111 podman[65341]: 2024-04-22 01:17:54.275822319 +0000 UTC m=+5.638142449 container died 026b18b63421f89043c0b0bf3a063d3b2b911a8aa7d5a4f08ffac0ab31760e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:54.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:54.417+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:54.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:54.417+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:54.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:54.417+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:54.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:54.417+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:54.895 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:54 smithi111 podman[65341]: 2024-04-22 01:17:54.68407141 +0000 UTC m=+6.046391536 container cleanup 026b18b63421f89043c0b0bf3a063d3b2b911a8aa7d5a4f08ffac0ab31760e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T01:17:54.895 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:54 smithi111 bash[65341]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7 2024-04-22T01:17:54.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:54 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:54.749+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:54.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:54 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:54.749+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:54.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:54 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:54.749+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:54.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:54 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:54.749+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:54.896 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:54 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:54.749+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:17:54.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:54.442+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:54.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:54.442+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:54.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:54.442+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:54.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:54.442+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:54.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:54 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:54.442+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:55.251 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:54 smithi111 podman[65358]: 2024-04-22 01:17:54.935302919 +0000 UTC m=+0.655140154 container remove 026b18b63421f89043c0b0bf3a063d3b2b911a8aa7d5a4f08ffac0ab31760e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T01:17:55.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:54.808+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:55.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:54.808+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:55.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:54.808+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:55.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:54 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:54.808+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:55.537 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:55 smithi111 podman[65448]: 2024-04-22 01:17:55.155309852 +0000 UTC m=+0.017715778 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:17:55.537 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:55 smithi111 podman[65448]: 2024-04-22 01:17:55.269570528 +0000 UTC m=+0.131976481 container create 7e63a1e1c7ada3ebb395ea6d2dd9b72a3ad873f068c582ce13dd5d5e31415f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T01:17:55.537 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:55 smithi111 podman[65448]: 2024-04-22 01:17:55.538721765 +0000 UTC m=+0.401127672 container init 7e63a1e1c7ada3ebb395ea6d2dd9b72a3ad873f068c582ce13dd5d5e31415f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T01:17:55.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:55.421+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:55.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:55.421+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:55.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:55.421+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:55.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:55.421+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:55 smithi111 podman[65448]: 2024-04-22 01:17:55.541993022 +0000 UTC m=+0.404398932 container start 7e63a1e1c7ada3ebb395ea6d2dd9b72a3ad873f068c582ce13dd5d5e31415f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:55 smithi111 podman[65448]: 2024-04-22 01:17:55.638629528 +0000 UTC m=+0.501035443 container attach 7e63a1e1c7ada3ebb395ea6d2dd9b72a3ad873f068c582ce13dd5d5e31415f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:55 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:55.780+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:55 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:55.780+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:55 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:55.780+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:55 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:55.780+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:55.827 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:55 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:55.780+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:17:55.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:55 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:55.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:55.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:55 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:55.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:55.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:55 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:55.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:55.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:55 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:55.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:55.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:55 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:55.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:56.115 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 01:17:55 smithi111 podman[65448]: 2024-04-22 01:17:55.82852213 +0000 UTC m=+0.690928036 container died 7e63a1e1c7ada3ebb395ea6d2dd9b72a3ad873f068c582ce13dd5d5e31415f00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T01:17:56.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:55.822+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:56.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:55.822+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:56.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:55.822+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:56.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:55 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:55.822+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:56.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T01:17:56.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:17:56.484 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T01:17:56.484 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-22T01:17:56.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.8 2024-04-22T01:17:56.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.405+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:56.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.405+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:56.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.405+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:56.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.405+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:56.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 systemd[1]: Stopping Ceph osd.8 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:17:56.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:56 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:56.486+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:56.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:56 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:56.486+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:56.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:56 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:56.486+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:56.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:56 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:56.486+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:56.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:56 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:56.486+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:57.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:56.740+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:57.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:56.740+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:57.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:56.740+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:57.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:56.740+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:57.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:56.740+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:17:57.266 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.807+0000 7f5b3e328700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:17:57.266 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.807+0000 7f5b3e328700 -1 osd.8 97 *** Got signal Terminated *** 2024-04-22T01:17:57.266 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:56.807+0000 7f5b3e328700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:17:57.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:56.829+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:57.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:56.829+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:57.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:56.829+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:57.267 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:56 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:56.829+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:57.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:57.400+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:57.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:57.400+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:57.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:57.400+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:57.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:57.400+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:57.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:57 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:57.782+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:57.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:57 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:57.782+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:57.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:57 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:57.782+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:57.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:57 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:57.782+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:57.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:57 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:57.782+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:17:57.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:57 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:57.495+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:57.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:57 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:57.495+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:57.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:57 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:57.495+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:57.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:57 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:57.495+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:57.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:57 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:57.495+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:58.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:57.861+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:58.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:57.861+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:58.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:57.861+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:58.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:57 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:57.861+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:58.549 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:58.403+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:58.549 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:58.403+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:58.549 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:58.403+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:58.549 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:58.403+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:58.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:58 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:58.467+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:58.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:58 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:58.467+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:58.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:58 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:58.467+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:58.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:58 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:58.467+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:58.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:58 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:58.467+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:59.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:58 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:58.815+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:17:59.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:58 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:58.815+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:17:59.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:58 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:58.815+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:17:59.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:58 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:58.815+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:17:59.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:58 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:58.815+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:17:59.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:58 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:58.815+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:17:59.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:58.817+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:17:59.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:58.817+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:17:59.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:58.818+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:17:59.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:58 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:58.818+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:17:59.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:59.420+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:17:59.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:59.420+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:17:59.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:59.420+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:17:59.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:59.420+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:17:59.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:17:59.420+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:34.110920+0000 front 2024-04-22T01:17:34.110898+0000 (oldest deadline 2024-04-22T01:17:59.410898+0000) 2024-04-22T01:17:59.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:59.477+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:17:59.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:59.477+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:17:59.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:59.477+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:17:59.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:59.477+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:17:59.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:59.477+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:17:59.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:17:59 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:17:59.477+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:00.059 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:59 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:59.802+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:00.059 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:59 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:59.802+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:00.059 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:59 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:59.802+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:00.059 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:59 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:59.802+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:00.059 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:59 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:59.802+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:00.059 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:17:59 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:17:59.802+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:00.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:59.833+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:00.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:59.833+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:00.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:59.833+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:00.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:59.833+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:00.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:17:59 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:17:59.833+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:00.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:00.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:18:00.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:00.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:18:00.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:00.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:18:00.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:00.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:18:00.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:00.379+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:34.110920+0000 front 2024-04-22T01:17:34.110898+0000 (oldest deadline 2024-04-22T01:17:59.410898+0000) 2024-04-22T01:18:00.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:00.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:00.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:00.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:00.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:00.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:00.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:00.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:00.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:00.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:00.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:00 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:00.479+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:01.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:00.774+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:01.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:00.774+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:01.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:00.774+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:01.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:00.774+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:01.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:00.774+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:01.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:00 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:00.774+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:01.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:00.819+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:01.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:00.819+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:01.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:00.819+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:01.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:00.819+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:01.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:00.819+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:01.765 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:01.363+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.104527+0000 front 2024-04-22T01:17:04.104635+0000 (oldest deadline 2024-04-22T01:17:27.004613+0000) 2024-04-22T01:18:01.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:01.363+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.606352+0000 front 2024-04-22T01:17:11.606380+0000 (oldest deadline 2024-04-22T01:17:34.506508+0000) 2024-04-22T01:18:01.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:01.363+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.808568+0000 front 2024-04-22T01:17:20.808701+0000 (oldest deadline 2024-04-22T01:17:43.108435+0000) 2024-04-22T01:18:01.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:01.363+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.310064+0000 front 2024-04-22T01:17:28.310109+0000 (oldest deadline 2024-04-22T01:17:52.409898+0000) 2024-04-22T01:18:01.766 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8[50737]: 2024-04-22T01:18:01.363+0000 7f5b36914700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:34.110920+0000 front 2024-04-22T01:17:34.110898+0000 (oldest deadline 2024-04-22T01:17:59.410898+0000) 2024-04-22T01:18:01.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:01 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:01.478+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:01.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:01 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:01.478+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:01.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:01 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:01.478+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:01.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:01 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:01.478+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:01.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:01 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:01.478+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:01.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:01 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:01.478+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:02.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:01 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:01.727+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:02.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:01 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:01.727+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:02.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:01 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:01.727+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:02.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:01 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:01.727+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:02.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:01 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:01.727+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:02.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:01 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:01.727+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:02.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:01.805+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:02.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:01.805+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:02.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:01.805+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:02.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:01.805+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:02.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:01 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:01.805+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:02.566 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:02 smithi203 podman[67047]: 2024-04-22 01:18:02.277864067 +0000 UTC m=+5.587864416 container died b52cf94bc72029140b46e3acb2b97adb199c62d83a29e895b559e7f34eb107d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:18:02.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:02 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:02.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:02.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:02 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:02.482+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:02.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:02 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:02.482+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:02.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:02 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:02.482+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:02.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:02 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:02.482+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:02.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:02 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:02.482+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:02.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:02 smithi203 podman[67047]: 2024-04-22 01:18:02.667004519 +0000 UTC m=+5.977004867 container cleanup b52cf94bc72029140b46e3acb2b97adb199c62d83a29e895b559e7f34eb107d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T01:18:02.908 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:02 smithi203 bash[67047]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8 2024-04-22T01:18:02.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:02 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:02.782+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:02.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:02 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:02.782+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:02.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:02 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:02.782+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:02.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:02 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:02.782+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:02.909 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:02 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:02.782+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:03.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:02 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:02.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:03.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:02 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:02.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:03.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:02 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:02.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:03.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:02 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:02.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:03.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:02 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:02.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:03.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:02 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:02.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:03.233 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:02 smithi203 podman[67075]: 2024-04-22 01:18:02.910115164 +0000 UTC m=+0.627271205 container remove b52cf94bc72029140b46e3acb2b97adb199c62d83a29e895b559e7f34eb107d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T01:18:03.515 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:03 smithi203 podman[67168]: 2024-04-22 01:18:03.137678992 +0000 UTC m=+0.017988667 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:18:03.516 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:03 smithi203 podman[67168]: 2024-04-22 01:18:03.26984981 +0000 UTC m=+0.150159529 container create 7baeda460775aad753ddf4b3c688912d66bdbab5f94c742fd6de41c7ff086c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T01:18:03.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:03 smithi203 podman[67168]: 2024-04-22 01:18:03.548170529 +0000 UTC m=+0.428480206 container init 7baeda460775aad753ddf4b3c688912d66bdbab5f94c742fd6de41c7ff086c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T01:18:03.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:03 smithi203 podman[67168]: 2024-04-22 01:18:03.551194134 +0000 UTC m=+0.431503813 container start 7baeda460775aad753ddf4b3c688912d66bdbab5f94c742fd6de41c7ff086c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T01:18:03.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:03 smithi203 podman[67168]: 2024-04-22 01:18:03.665050618 +0000 UTC m=+0.545360299 container attach 7baeda460775aad753ddf4b3c688912d66bdbab5f94c742fd6de41c7ff086c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T01:18:03.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:03.484+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:03.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:03.484+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:03.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:03.484+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:03.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:03.484+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:03.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:03.484+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:03.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:03 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:03.484+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:04.125 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 01:18:03 smithi203 podman[67168]: 2024-04-22 01:18:03.844015532 +0000 UTC m=+0.724325210 container died 7baeda460775aad753ddf4b3c688912d66bdbab5f94c742fd6de41c7ff086c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-22T01:18:04.125 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:03 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:03.770+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:04.125 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:03 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:03.770+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:04.125 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:03 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:03.770+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:04.125 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:03 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:03.770+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:04.125 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:03 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:03.770+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:04.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:03 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:03.770+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:04.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:03 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:03.770+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:04.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:03 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:03.770+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:04.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:03 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:03.770+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:04.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:03 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:03.771+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:04.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:03 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:03.771+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:04.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.8.service' 2024-04-22T01:18:04.437 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:04.438 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-22T01:18:04.438 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-22T01:18:04.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.9 2024-04-22T01:18:04.754 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.440+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.440+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.440+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.440+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.440+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.440+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 systemd[1]: Stopping Ceph osd.9 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.754+0000 7f9ae80f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.754+0000 7f9ae80f8700 -1 osd.9 97 *** Got signal Terminated *** 2024-04-22T01:18:04.755 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:04 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:04.754+0000 7f9ae80f8700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:18:05.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:04 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:04.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:05.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:04 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:04.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:05.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:04 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:04.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:05.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:04 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:04.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:05.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:04 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:04.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:05.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:04 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:04.750+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:05.265 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:04.769+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:05.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:04.769+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:05.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:04.769+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:05.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:04.769+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:05.266 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:04.769+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:05.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:05 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:05.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:05.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:05 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:05.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:05.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:05 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:05.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:05.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:05 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:05.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:05.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:05 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:05.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:05.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:05 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:05.481+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:06.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:05 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:05.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:06.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:05 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:05.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:06.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:05 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:05.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:06.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:05 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:05.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:06.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:05 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:05.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:06.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:05 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:05.712+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:06.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:05 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:05.712+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:06.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:05 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:05.712+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:06.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:05 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:05.712+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:06.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:05 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:05.712+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:06.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:05 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:05.712+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:06.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:06 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:06.529+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:06.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:06 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:06.529+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:06.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:06 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:06.529+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:06.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:06 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:06.529+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:06.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:06 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:06.529+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:06.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:06 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:06.529+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:07.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:06 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:06.690+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:07.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:06 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:06.690+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:07.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:06 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:06.690+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:07.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:06 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:06.690+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:07.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:06 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:06.690+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:07.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:06 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:06.690+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:07.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:06 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:06.759+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:07.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:06 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:06.759+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:07.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:06 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:06.759+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:07.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:06 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:06.759+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:07.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:06 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:06.759+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:07.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:06 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:06.759+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:07.904 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:07 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:07.567+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:07.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:07 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:07.567+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:07.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:07 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:07.567+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:07.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:07 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:07.567+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:07.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:07 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:07.567+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:07.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:07 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:07.567+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:08.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:07 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:07.701+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:08.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:07 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:07.701+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:08.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:07 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:07.701+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:08.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:07 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:07.701+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:08.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:07 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:07.701+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:08.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:07 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:07.701+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:08.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:07 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:07.797+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:08.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:07 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:07.797+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:08.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:07 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:07.797+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:08.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:07 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:07.797+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:08.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:07 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:07.797+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:08.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:07 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:07.797+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:08.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:08.610+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:08.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:08.610+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:08.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:08.610+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:08.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:08.610+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:08.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:08.610+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:08.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:08 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:08.610+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:09.017 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:08.722+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:09.018 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:08.722+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:09.018 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:08.722+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:09.018 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:08.722+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:09.018 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:08.722+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:09.018 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:08.722+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:09.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:08 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:08.820+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:09.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:08 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:08.820+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:09.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:08 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:08.820+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:09.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:08 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:08.820+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:09.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:08 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:08.820+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:09.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:08 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:08.820+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:09.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:09.579+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.159984+0000 front 2024-04-22T01:16:55.159952+0000 (oldest deadline 2024-04-22T01:17:19.260095+0000) 2024-04-22T01:18:09.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:09.579+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:05.161110+0000 front 2024-04-22T01:17:05.161068+0000 (oldest deadline 2024-04-22T01:17:29.861202+0000) 2024-04-22T01:18:09.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:09.579+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:11.562242+0000 front 2024-04-22T01:17:11.562255+0000 (oldest deadline 2024-04-22T01:17:34.462210+0000) 2024-04-22T01:18:09.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:09.579+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:21.463473+0000 front 2024-04-22T01:17:21.463504+0000 (oldest deadline 2024-04-22T01:17:43.763474+0000) 2024-04-22T01:18:09.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:09.579+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:27.765022+0000 front 2024-04-22T01:17:27.765156+0000 (oldest deadline 2024-04-22T01:17:50.664950+0000) 2024-04-22T01:18:09.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:09 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9[62473]: 2024-04-22T01:18:09.579+0000 7f9ae46ec700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:35.767377+0000 front 2024-04-22T01:17:35.767162+0000 (oldest deadline 2024-04-22T01:17:58.667015+0000) 2024-04-22T01:18:10.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:09 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:09.752+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:10.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:09 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:09.752+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:10.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:09 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:09.752+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:10.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:09 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:09.752+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:10.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:09 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:09.752+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:10.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:09 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:09.752+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:10.135 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:09 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:09.860+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:10.404 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:10 smithi038 podman[89792]: 2024-04-22 01:18:10.099384687 +0000 UTC m=+5.455013808 container died 4cbf76feea3df14cc9429f0036f15de864b6d7c3f1fc63336612c03001cf74cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T01:18:10.746 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:10 smithi038 podman[89792]: 2024-04-22 01:18:10.496495268 +0000 UTC m=+5.852124395 container cleanup 4cbf76feea3df14cc9429f0036f15de864b6d7c3f1fc63336612c03001cf74cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T01:18:10.746 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:10 smithi038 bash[89792]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9 2024-04-22T01:18:11.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:10 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:10.743+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:11.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:10 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:10.743+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:11.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:10 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:10.743+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:11.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:10 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:10.743+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:11.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:10 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:10.743+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:11.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:10 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:10.743+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:11.080 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:10 smithi038 podman[89809]: 2024-04-22 01:18:10.747780118 +0000 UTC m=+0.647149991 container remove 4cbf76feea3df14cc9429f0036f15de864b6d7c3f1fc63336612c03001cf74cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T01:18:11.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:11.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:11.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:11.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:11.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:11.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:11.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:10 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:10.839+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:11.333 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:11 smithi038 podman[89898]: 2024-04-22 01:18:10.984979765 +0000 UTC m=+0.018794039 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:18:11.333 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:11 smithi038 podman[89898]: 2024-04-22 01:18:11.090212962 +0000 UTC m=+0.124027220 container create f05de4dc6095d83a30ca320ebe288d0b7950e44870c5c61cbdc1b54f331fa920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T01:18:11.333 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:11 smithi038 podman[89898]: 2024-04-22 01:18:11.334569451 +0000 UTC m=+0.368383710 container init f05de4dc6095d83a30ca320ebe288d0b7950e44870c5c61cbdc1b54f331fa920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_CLEAN=True) 2024-04-22T01:18:11.641 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:11 smithi038 podman[89898]: 2024-04-22 01:18:11.338196194 +0000 UTC m=+0.372010462 container start f05de4dc6095d83a30ca320ebe288d0b7950e44870c5c61cbdc1b54f331fa920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8) 2024-04-22T01:18:11.641 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:11 smithi038 podman[89898]: 2024-04-22 01:18:11.434912638 +0000 UTC m=+0.468726897 container attach f05de4dc6095d83a30ca320ebe288d0b7950e44870c5c61cbdc1b54f331fa920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T01:18:11.905 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 01:18:11 smithi038 podman[89898]: 2024-04-22 01:18:11.642522729 +0000 UTC m=+0.676336980 container died f05de4dc6095d83a30ca320ebe288d0b7950e44870c5c61cbdc1b54f331fa920 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T01:18:12.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:11.733+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:12.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:11.733+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:12.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:11.733+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:12.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:11.733+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:12.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:11.733+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:12.017 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:11 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:11.733+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:12.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:12.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:12.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:12.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:12.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:12.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:12.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:11 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:11.852+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:12.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.9.service' 2024-04-22T01:18:12.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:12.262 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-22T01:18:12.263 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-22T01:18:12.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.10 2024-04-22T01:18:12.560 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 systemd[1]: Stopping Ceph osd.10 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:12.560 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.560+0000 7fc9ff3ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:12.857 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.560+0000 7fc9ff3ce700 -1 osd.10 97 *** Got signal Terminated *** 2024-04-22T01:18:12.857 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.560+0000 7fc9ff3ce700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:18:13.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:12.717+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:13.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:12.717+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:13.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:12.717+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:13.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:12.717+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:13.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:12.717+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:13.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:12.717+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:13.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:12 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:12.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:14.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:13 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:13.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:14.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:13 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:13.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:14.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:13 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:13.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:14.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:13 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:13.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:14.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:13 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:13.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:14.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:13 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:13.735+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:14.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:14.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:14.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:14.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:14.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:14.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:14.146 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:13 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:13.857+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:15.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:14 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:14.713+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:15.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:14 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:14.713+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:15.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:14 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:14.713+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:15.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:14 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:14.713+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:15.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:14 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:14.713+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:15.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:14 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:14.713+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:15.145 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:14 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:14.859+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:16.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:15 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:15.670+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:16.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:15 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:15.670+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:16.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:15 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:15.670+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:16.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:15 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:15.670+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:16.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:15 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:15.670+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:16.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:15 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:15.670+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:16.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:15 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:15.907+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:17.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:16 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:16.647+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:17.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:16 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:16.647+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:17.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:16 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:16.647+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:17.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:16 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:16.647+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:17.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:16 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:16.647+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:17.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:16 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:16.647+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:55.117960+0000 front 2024-04-22T01:16:55.117923+0000 (oldest deadline 2024-04-22T01:17:18.017837+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:02.619603+0000 front 2024-04-22T01:17:02.619532+0000 (oldest deadline 2024-04-22T01:17:26.620205+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.121273+0000 front 2024-04-22T01:17:10.121382+0000 (oldest deadline 2024-04-22T01:17:35.421341+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-04-22T01:17:20.022972+0000 front 2024-04-22T01:17:20.022940+0000 (oldest deadline 2024-04-22T01:17:45.322902+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.724426+0000 front 2024-04-22T01:17:28.724455+0000 (oldest deadline 2024-04-22T01:17:52.824533+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:32.825285+0000 front 2024-04-22T01:17:32.825304+0000 (oldest deadline 2024-04-22T01:17:58.124963+0000) 2024-04-22T01:18:17.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:16 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10[55960]: 2024-04-22T01:18:16.915+0000 7fc9fb9c2700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:45.026825+0000 front 2024-04-22T01:17:45.026803+0000 (oldest deadline 2024-04-22T01:18:09.726612+0000) 2024-04-22T01:18:18.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:18.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:18.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:18.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:18.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:18.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:18.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:17 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:17.627+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:18.395 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:18 smithi111 podman[65567]: 2024-04-22 01:18:18.080805153 +0000 UTC m=+5.624280645 container died 8eb9d401545ff7a90c45c0298b374b01d1ee8b5fc5ba7d7bb56bc7149c194c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:18:18.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:18 smithi111 podman[65567]: 2024-04-22 01:18:18.401986764 +0000 UTC m=+5.945462265 container cleanup 8eb9d401545ff7a90c45c0298b374b01d1ee8b5fc5ba7d7bb56bc7149c194c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=) 2024-04-22T01:18:18.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:18 smithi111 bash[65567]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10 2024-04-22T01:18:18.895 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:18 smithi111 podman[65583]: 2024-04-22 01:18:18.620093007 +0000 UTC m=+0.533352447 container remove 8eb9d401545ff7a90c45c0298b374b01d1ee8b5fc5ba7d7bb56bc7149c194c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:19.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:18 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:18.608+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:19.308 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:19 smithi111 podman[65675]: 2024-04-22 01:18:18.962174595 +0000 UTC m=+0.017676213 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:18:19.308 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:19 smithi111 podman[65675]: 2024-04-22 01:18:19.095979462 +0000 UTC m=+0.151481080 container create 5d6aa1bc9af48b1a3bb266cb91af4f55bf3cb7066cdb7a657436ca0ff15a98dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:18:19.308 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:19 smithi111 podman[65675]: 2024-04-22 01:18:19.306602662 +0000 UTC m=+0.362104280 container init 5d6aa1bc9af48b1a3bb266cb91af4f55bf3cb7066cdb7a657436ca0ff15a98dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:18:19.308 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:19 smithi111 podman[65675]: 2024-04-22 01:18:19.309625741 +0000 UTC m=+0.365127360 container start 5d6aa1bc9af48b1a3bb266cb91af4f55bf3cb7066cdb7a657436ca0ff15a98dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T01:18:19.597 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:19 smithi111 podman[65675]: 2024-04-22 01:18:19.414918302 +0000 UTC m=+0.470419932 container attach 5d6aa1bc9af48b1a3bb266cb91af4f55bf3cb7066cdb7a657436ca0ff15a98dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-22T01:18:19.891 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 01:18:19 smithi111 podman[65675]: 2024-04-22 01:18:19.598532475 +0000 UTC m=+0.654034094 container died 5d6aa1bc9af48b1a3bb266cb91af4f55bf3cb7066cdb7a657436ca0ff15a98dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:20.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:19 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:19.622+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:20.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.10.service' 2024-04-22T01:18:20.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:20.233 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-22T01:18:20.233 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-22T01:18:20.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.11 2024-04-22T01:18:20.516 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 systemd[1]: Stopping Ceph osd.11 for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.554+0000 7f3526d54700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.554+0000 7f3526d54700 -1 osd.11 97 *** Got signal Terminated *** 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.554+0000 7f3526d54700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:21.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:21.017 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:21.017 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:21.017 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:20 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:20.605+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:22.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:21 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:21.598+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:23.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:22 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:22.570+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:24.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:23 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:23.566+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-04-22T01:17:57.615346+0000 front 2024-04-22T01:17:57.615365+0000 (oldest deadline 2024-04-22T01:18:23.515353+0000) 2024-04-22T01:18:25.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:25.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:24 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:24.593+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-04-22T01:17:57.615346+0000 front 2024-04-22T01:17:57.615365+0000 (oldest deadline 2024-04-22T01:18:23.515353+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T01:16:56.705884+0000 front 2024-04-22T01:16:56.705752+0000 (oldest deadline 2024-04-22T01:17:21.306245+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T01:17:04.807221+0000 front 2024-04-22T01:17:04.807069+0000 (oldest deadline 2024-04-22T01:17:25.907162+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T01:17:10.008151+0000 front 2024-04-22T01:17:10.008231+0000 (oldest deadline 2024-04-22T01:17:35.308050+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6812 osd.4 since back 2024-04-22T01:17:28.710614+0000 front 2024-04-22T01:17:28.710706+0000 (oldest deadline 2024-04-22T01:17:51.010450+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-22T01:17:36.211846+0000 front 2024-04-22T01:17:36.211919+0000 (oldest deadline 2024-04-22T01:17:59.111913+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-04-22T01:17:44.214014+0000 front 2024-04-22T01:17:44.213912+0000 (oldest deadline 2024-04-22T01:18:06.513797+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.111:6820 osd.7 since back 2024-04-22T01:17:48.214572+0000 front 2024-04-22T01:17:48.214644+0000 (oldest deadline 2024-04-22T01:18:17.615020+0000) 2024-04-22T01:18:26.016 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:25 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11[57503]: 2024-04-22T01:18:25.554+0000 7f351f340700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-04-22T01:17:57.615346+0000 front 2024-04-22T01:17:57.615365+0000 (oldest deadline 2024-04-22T01:18:23.515353+0000) 2024-04-22T01:18:26.297 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:26 smithi203 podman[67272]: 2024-04-22 01:18:26.043895722 +0000 UTC m=+5.617448603 container died 9896b87a5854fb9753c41a6720ca97a3311aa015944edf3b6960c536815ed852 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T01:18:26.605 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:26 smithi203 podman[67272]: 2024-04-22 01:18:26.396603772 +0000 UTC m=+5.970156643 container cleanup 9896b87a5854fb9753c41a6720ca97a3311aa015944edf3b6960c536815ed852 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-22T01:18:26.605 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:26 smithi203 bash[67272]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11 2024-04-22T01:18:27.015 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:26 smithi203 podman[67291]: 2024-04-22 01:18:26.606338676 +0000 UTC m=+0.554132290 container remove 9896b87a5854fb9753c41a6720ca97a3311aa015944edf3b6960c536815ed852 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T01:18:27.326 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:27 smithi203 podman[67382]: 2024-04-22 01:18:26.92762567 +0000 UTC m=+0.017998169 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T01:18:27.326 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:27 smithi203 podman[67382]: 2024-04-22 01:18:27.041282054 +0000 UTC m=+0.131654539 container create 78acb6e832bd13e2b6212ce9d485ad0ee6279af6ea9dd2e4b159c792283cbda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T01:18:27.326 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:27 smithi203 podman[67382]: 2024-04-22 01:18:27.261108085 +0000 UTC m=+0.351480572 container init 78acb6e832bd13e2b6212ce9d485ad0ee6279af6ea9dd2e4b159c792283cbda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:18:27.326 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:27 smithi203 podman[67382]: 2024-04-22 01:18:27.264217279 +0000 UTC m=+0.354589774 container start 78acb6e832bd13e2b6212ce9d485ad0ee6279af6ea9dd2e4b159c792283cbda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-22T01:18:27.326 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:27 smithi203 podman[67382]: 2024-04-22 01:18:27.327276465 +0000 UTC m=+0.417648952 container attach 78acb6e832bd13e2b6212ce9d485ad0ee6279af6ea9dd2e4b159c792283cbda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T01:18:27.765 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:27 smithi203 podman[67382]: 2024-04-22 01:18:27.554886594 +0000 UTC m=+0.645259074 container died 78acb6e832bd13e2b6212ce9d485ad0ee6279af6ea9dd2e4b159c792283cbda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T01:18:28.116 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 01:18:28 smithi203 podman[67408]: 2024-04-22 01:18:28.106211633 +0000 UTC m=+0.546709893 container remove 78acb6e832bd13e2b6212ce9d485ad0ee6279af6ea9dd2e4b159c792283cbda3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8) 2024-04-22T01:18:28.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@osd.11.service' 2024-04-22T01:18:28.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:28.159 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-22T01:18:28.159 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-22T01:18:28.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.a 2024-04-22T01:18:28.482 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:18:28 smithi038 systemd[1]: Stopping Ceph mds.a for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:28.904 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:18:28 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a[69206]: 2024-04-22T01:18:28.481+0000 7fc10e542700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:28.904 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:18:28 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a[69206]: 2024-04-22T01:18:28.481+0000 7fc10e542700 -1 mds.a *** got signal Terminated *** 2024-04-22T01:18:31.257 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:18:30 smithi038 podman[90016]: 2024-04-22 01:18:30.987957839 +0000 UTC m=+2.626543703 container died cccf62f2b37b0ae8338f8df1e7ea9ce5db62a3ea23f83f1d773c0d47fd650ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:18:31.599 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:18:31 smithi038 podman[90016]: 2024-04-22 01:18:31.340963888 +0000 UTC m=+2.979549804 container cleanup cccf62f2b37b0ae8338f8df1e7ea9ce5db62a3ea23f83f1d773c0d47fd650ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T01:18:31.599 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 01:18:31 smithi038 bash[90016]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-a 2024-04-22T01:18:31.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.a.service' 2024-04-22T01:18:31.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:31.764 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-22T01:18:31.764 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-22T01:18:31.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.d 2024-04-22T01:18:31.904 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:18:31 smithi038 systemd[1]: Stopping Ceph mds.d for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:32.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:18:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d[68236]: 2024-04-22T01:18:32.077+0000 7f56801ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:32.405 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:18:32 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d[68236]: 2024-04-22T01:18:32.077+0000 7f56801ca700 -1 mds.d *** got signal Terminated *** 2024-04-22T01:18:35.494 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:18:35 smithi038 podman[90135]: 2024-04-22 01:18:35.220680711 +0000 UTC m=+3.231794936 container died f1bf8db94a19bc683fc11e3b12aef7a10dee8c7b76903869ef4101fffd71ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:18:35.877 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:18:35 smithi038 podman[90135]: 2024-04-22 01:18:35.602839648 +0000 UTC m=+3.613953868 container cleanup f1bf8db94a19bc683fc11e3b12aef7a10dee8c7b76903869ef4101fffd71ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d, RELEASE=squid-820a2f8, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T01:18:35.878 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 01:18:35 smithi038 bash[90135]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-d 2024-04-22T01:18:36.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.d.service' 2024-04-22T01:18:36.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:36.062 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-22T01:18:36.062 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-22T01:18:36.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.g 2024-04-22T01:18:36.404 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:36 smithi038 systemd[1]: Stopping Ceph mds.g for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:36.405 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:36 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g[68847]: 2024-04-22T01:18:36.381+0000 7fb548fe1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:36.405 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:36 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g[68847]: 2024-04-22T01:18:36.381+0000 7fb548fe1700 -1 mds.g *** got signal Terminated *** 2024-04-22T01:18:39.356 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:38 smithi038 podman[90254]: 2024-04-22 01:18:38.99256544 +0000 UTC m=+2.713126375 container died fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T01:18:39.607 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:39 smithi038 podman[90254]: 2024-04-22 01:18:39.431839239 +0000 UTC m=+3.152400173 container cleanup fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:18:39.607 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:39 smithi038 bash[90254]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g 2024-04-22T01:18:39.863 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 01:18:39 smithi038 podman[90269]: 2024-04-22 01:18:39.608132217 +0000 UTC m=+0.613817589 container remove fc39d84caa4e4dba25c7ea585dd4b3804adf56f533cc2ff6ee7dcb4df6cc6c8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T01:18:39.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.g.service' 2024-04-22T01:18:39.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:39.899 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-22T01:18:39.899 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-22T01:18:39.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.j 2024-04-22T01:18:40.155 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:39 smithi038 systemd[1]: Stopping Ceph mds.j for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:40.654 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j[67842]: 2024-04-22T01:18:40.227+0000 7fb1486b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:40.654 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:40 smithi038 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j[67842]: 2024-04-22T01:18:40.227+0000 7fb1486b8700 -1 mds.j *** got signal Terminated *** 2024-04-22T01:18:44.510 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:44 smithi038 podman[90376]: 2024-04-22 01:18:44.232870647 +0000 UTC m=+4.117133965 container died e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T01:18:44.768 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:44 smithi038 podman[90376]: 2024-04-22 01:18:44.585649025 +0000 UTC m=+4.469912298 container cleanup e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T01:18:44.769 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:44 smithi038 bash[90376]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j 2024-04-22T01:18:45.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.j.service' 2024-04-22T01:18:45.023 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:44 smithi038 podman[90390]: 2024-04-22 01:18:44.769896997 +0000 UTC m=+0.526309867 container remove e0f2e6ad64c3800eebf226d4a69a69e8f71f16008481dd838892880697a28dc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T01:18:45.024 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:45 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.j.service: Deactivated successfully. 2024-04-22T01:18:45.024 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:45 smithi038 systemd[1]: Stopped Ceph mds.j for caf31b2e-0043-11ef-bc93-c7b262605968. 2024-04-22T01:18:45.024 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 01:18:45 smithi038 systemd[1]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.017s CPU time. 2024-04-22T01:18:45.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:45.035 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-22T01:18:45.035 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-22T01:18:45.035 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.b 2024-04-22T01:18:45.395 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:18:45 smithi111 systemd[1]: Stopping Ceph mds.b for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:45.651 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:18:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b[61609]: 2024-04-22T01:18:45.476+0000 7f9b0648d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:45.651 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:18:45 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b[61609]: 2024-04-22T01:18:45.476+0000 7f9b0648d700 -1 mds.b *** got signal Terminated *** 2024-04-22T01:18:47.292 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:18:46 smithi111 podman[65782]: 2024-04-22 01:18:46.983019829 +0000 UTC m=+1.760504038 container died 4fee09027b0ad91ef22acd4573204148c7a949d026ce6b234bc27e5b5b95c36d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T01:18:47.626 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:18:47 smithi111 podman[65782]: 2024-04-22 01:18:47.384526741 +0000 UTC m=+2.162010942 container cleanup 4fee09027b0ad91ef22acd4573204148c7a949d026ce6b234bc27e5b5b95c36d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-22T01:18:47.682 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 01:18:47 smithi111 bash[65782]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-b 2024-04-22T01:18:47.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.b.service' 2024-04-22T01:18:47.787 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:47.787 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-22T01:18:47.787 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-22T01:18:47.788 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.e 2024-04-22T01:18:47.895 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:18:47 smithi111 systemd[1]: Stopping Ceph mds.e for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:48.394 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:18:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e[60998]: 2024-04-22T01:18:48.113+0000 7f7284996700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:48.395 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:18:48 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e[60998]: 2024-04-22T01:18:48.113+0000 7f7284996700 -1 mds.e *** got signal Terminated *** 2024-04-22T01:18:51.296 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:18:50 smithi111 podman[65915]: 2024-04-22 01:18:50.983233209 +0000 UTC m=+2.980068920 container died cb02eaa3192a073f548e831ba4aacc89b766642d1dd707652f76b254846e2b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-22T01:18:51.633 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:18:51 smithi111 podman[65915]: 2024-04-22 01:18:51.379855077 +0000 UTC m=+3.376690784 container cleanup cb02eaa3192a073f548e831ba4aacc89b766642d1dd707652f76b254846e2b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T01:18:51.633 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 01:18:51 smithi111 bash[65915]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-e 2024-04-22T01:18:51.764 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.e.service' 2024-04-22T01:18:51.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:51.795 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-22T01:18:51.796 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-22T01:18:51.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.h 2024-04-22T01:18:51.895 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:18:51 smithi111 systemd[1]: Stopping Ceph mds.h for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:52.395 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:18:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h[61299]: 2024-04-22T01:18:52.116+0000 7f6a9fa5f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:52.395 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:18:52 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h[61299]: 2024-04-22T01:18:52.116+0000 7f6a9fa5f700 -1 mds.h *** got signal Terminated *** 2024-04-22T01:18:55.291 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:18:54 smithi111 podman[66036]: 2024-04-22 01:18:54.997421613 +0000 UTC m=+2.988366337 container died a7209b95fa1783210cf1167f2040a52a74b16d255b803319e84a579bde2ce680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T01:18:55.616 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:18:55 smithi111 podman[66036]: 2024-04-22 01:18:55.366522651 +0000 UTC m=+3.357467375 container cleanup a7209b95fa1783210cf1167f2040a52a74b16d255b803319e84a579bde2ce680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-820a2f8) 2024-04-22T01:18:55.616 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 01:18:55 smithi111 bash[66036]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-h 2024-04-22T01:18:55.738 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.h.service' 2024-04-22T01:18:55.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:55.769 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-22T01:18:55.769 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-22T01:18:55.769 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.k 2024-04-22T01:18:55.895 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:18:55 smithi111 systemd[1]: Stopping Ceph mds.k for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:18:56.395 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:18:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k[60675]: 2024-04-22T01:18:56.068+0000 7fd5a32c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:18:56.395 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:18:56 smithi111 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k[60675]: 2024-04-22T01:18:56.068+0000 7fd5a32c2700 -1 mds.k *** got signal Terminated *** 2024-04-22T01:18:59.269 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:18:58 smithi111 podman[66157]: 2024-04-22 01:18:58.99621944 +0000 UTC m=+3.014218178 container died f82f4e5060e9093a43252a9bb32bbf936b7b408fe17030485dde11e22ca1ce32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T01:18:59.627 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:18:59 smithi111 podman[66157]: 2024-04-22 01:18:59.369566978 +0000 UTC m=+3.387565715 container cleanup f82f4e5060e9093a43252a9bb32bbf936b7b408fe17030485dde11e22ca1ce32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, io.buildah.version=1.33.5) 2024-04-22T01:18:59.627 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 01:18:59 smithi111 bash[66157]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-k 2024-04-22T01:18:59.791 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.k.service' 2024-04-22T01:18:59.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:18:59.823 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-22T01:18:59.823 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-22T01:18:59.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.c 2024-04-22T01:19:00.122 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:18:59 smithi203 systemd[1]: Stopping Ceph mds.c for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:19:00.382 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:19:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c[62939]: 2024-04-22T01:19:00.122+0000 7f18d8440700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:19:00.382 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:19:00 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c[62939]: 2024-04-22T01:19:00.122+0000 7f18d8440700 -1 mds.c *** got signal Terminated *** 2024-04-22T01:19:03.266 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:19:02 smithi203 podman[67501]: 2024-04-22 01:19:02.99003455 +0000 UTC m=+2.967480704 container died a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-820a2f8) 2024-04-22T01:19:03.641 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:19:03 smithi203 podman[67501]: 2024-04-22 01:19:03.27334766 +0000 UTC m=+3.250793858 container cleanup a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8) 2024-04-22T01:19:03.642 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:19:03 smithi203 bash[67501]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c 2024-04-22T01:19:03.642 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 01:19:03 smithi203 podman[67527]: 2024-04-22 01:19:03.516142691 +0000 UTC m=+0.522794193 container remove a2804210d0673f3f6639be465e2be9b17522e1d99e04a564cba3e6d1b9d7c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-c, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T01:19:03.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.c.service' 2024-04-22T01:19:03.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:19:03.692 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-22T01:19:03.692 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-22T01:19:03.692 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.f 2024-04-22T01:19:04.015 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:19:03 smithi203 systemd[1]: Stopping Ceph mds.f for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:19:04.515 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:19:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f[62633]: 2024-04-22T01:19:04.026+0000 7fd3abc94700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:19:04.516 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:19:04 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f[62633]: 2024-04-22T01:19:04.026+0000 7fd3abc94700 -1 mds.f *** got signal Terminated *** 2024-04-22T01:19:07.265 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:19:06 smithi203 podman[67633]: 2024-04-22 01:19:06.988070896 +0000 UTC m=+3.075736972 container died aaf036dce494b6a7919c96b5f2d0ef21685284ce9c2375cd4c5546e299a912a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T01:19:07.602 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:19:07 smithi203 podman[67633]: 2024-04-22 01:19:07.360550343 +0000 UTC m=+3.448216425 container cleanup aaf036dce494b6a7919c96b5f2d0ef21685284ce9c2375cd4c5546e299a912a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True) 2024-04-22T01:19:07.602 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 01:19:07 smithi203 bash[67633]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-f 2024-04-22T01:19:07.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.f.service' 2024-04-22T01:19:07.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:19:07.761 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-22T01:19:07.761 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-22T01:19:07.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.i 2024-04-22T01:19:08.015 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 01:19:07 smithi203 systemd[1]: Stopping Ceph mds.i for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:19:08.515 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 01:19:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-i[63248]: 2024-04-22T01:19:08.072+0000 7fe1cef70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:19:08.516 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 01:19:08 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-i[63248]: 2024-04-22T01:19:08.072+0000 7fe1cef70700 -1 mds.i *** got signal Terminated *** 2024-04-22T01:19:11.266 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 01:19:10 smithi203 podman[67753]: 2024-04-22 01:19:10.992954348 +0000 UTC m=+3.017875931 container died b20e95da79c9fa4b81407f68cbbf8f7bf4f1388c9a08d101ca3fd58e2c721536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-i, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T01:19:11.598 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 01:19:11 smithi203 podman[67753]: 2024-04-22 01:19:11.381421588 +0000 UTC m=+3.406343180 container cleanup b20e95da79c9fa4b81407f68cbbf8f7bf4f1388c9a08d101ca3fd58e2c721536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T01:19:11.598 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 01:19:11 smithi203 bash[67753]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-i 2024-04-22T01:19:11.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.i.service' 2024-04-22T01:19:11.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:19:11.749 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-22T01:19:11.749 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-22T01:19:11.749 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.l 2024-04-22T01:19:12.015 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:11 smithi203 systemd[1]: Stopping Ceph mds.l for caf31b2e-0043-11ef-bc93-c7b262605968... 2024-04-22T01:19:12.516 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l[62327]: 2024-04-22T01:19:12.068+0000 7fb7f14f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T01:19:12.516 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:12 smithi203 ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l[62327]: 2024-04-22T01:19:12.068+0000 7fb7f14f2700 -1 mds.l *** got signal Terminated *** 2024-04-22T01:19:15.252 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:14 smithi203 podman[67876]: 2024-04-22 01:19:14.987731817 +0000 UTC m=+3.024211407 container died e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T01:19:15.515 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:15 smithi203 podman[67876]: 2024-04-22 01:19:15.3188667 +0000 UTC m=+3.355346291 container cleanup e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T01:19:15.516 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:15 smithi203 bash[67876]: ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l 2024-04-22T01:19:15.516 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 01:19:15 smithi203 podman[67890]: 2024-04-22 01:19:15.495068438 +0000 UTC m=+0.503295235 container remove e0d05dfe669baf6fbd507afc4f02c3b85b9a947cc1f156a9a9f748630fe11576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-caf31b2e-0043-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T01:19:15.739 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-caf31b2e-0043-11ef-bc93-c7b262605968@mds.l.service' 2024-04-22T01:19:15.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T01:19:15.770 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-22T01:19:15.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T01:19:15.969 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:19:18.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T01:19:18.793 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:19:21.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T01:19:21.380 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:19:23.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:19:23.693 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:19:23.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T01:19:23.752 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T01:19:23.804 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi038/crash 2024-04-22T01:19:23.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash -- . 2024-04-22T01:19:23.846 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T01:19:23.846 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-04-22T01:19:23.848 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi111/crash 2024-04-22T01:19:23.850 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash -- . 2024-04-22T01:19:23.936 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T01:19:23.937 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-04-22T01:19:23.939 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi203/crash 2024-04-22T01:19:23.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash -- . 2024-04-22T01:19:23.978 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T01:19:23.978 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-04-22T01:19:23.979 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T01:19:23.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-22T01:19:24.016 INFO:tasks.cephadm:Compressing logs... 2024-04-22T01:19:24.017 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T01:19:24.060 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T01:19:24.062 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T01:19:24.085 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-22T01:19:24.085 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log 2024-04-22T01:19:24.085 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T01:19:24.097 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-22T01:19:24.097 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.x.log: 86.2%gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:gzip 89.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.a.log: -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log: 94.8% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 82.8% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T01:19:24.098 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:: No such file or directory 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-22T01:19:24.099 INFO:teuthology.orchestra.run.smithi203.stderr: 86.1% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T01:19:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T01:19:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.z.log: /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T01:19:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-22T01:19:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.6% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T01:19:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-22T01:19:24.100 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.5% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-22T01:19:24.103 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log 2024-04-22T01:19:24.103 INFO:teuthology.orchestra.run.smithi111.stderr:: No such file or directory 2024-04-22T01:19:24.103 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T01:19:24.103 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T01:19:24.103 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-22T01:19:24.103 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.1.log: 86.1%gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.h.log: /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.10.log: 90.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.8% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.e.log: 92.3% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T01:19:24.104 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T01:19:24.106 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T01:19:24.106 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-22T01:19:24.108 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-22T01:19:24.110 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.7%gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T01:19:24.110 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.9.log: -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-22T01:19:24.111 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T01:19:24.112 INFO:teuthology.orchestra.run.smithi038.stderr: 91.1% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T01:19:24.117 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-22T01:19:24.122 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-22T01:19:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T01:19:24.124 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T01:19:24.125 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cephadm.log: 94.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-22T01:19:24.125 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.l.log: 92.7% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-22T01:19:24.133 INFO:teuthology.orchestra.run.smithi038.stderr: 94.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-22T01:19:24.135 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T01:19:24.136 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-22T01:19:24.138 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 92.3% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T01:19:24.139 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-22T01:19:24.139 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T01:19:24.144 INFO:teuthology.orchestra.run.smithi203.stderr: 91.1% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T01:19:24.144 INFO:teuthology.orchestra.run.smithi111.stderr: 92.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-22T01:19:24.146 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-22T01:19:24.159 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-22T01:19:24.168 INFO:teuthology.orchestra.run.smithi111.stderr: 94.1% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-22T01:19:24.251 INFO:teuthology.orchestra.run.smithi038.stderr: 90.1% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T01:19:24.326 INFO:teuthology.orchestra.run.smithi203.stderr: 96.4% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-22T01:19:24.328 INFO:teuthology.orchestra.run.smithi111.stderr: 92.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T01:19:24.357 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T01:19:24.702 INFO:teuthology.orchestra.run.smithi038.stderr: 90.6% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T01:19:26.431 INFO:teuthology.orchestra.run.smithi111.stderr: 94.8% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-22T01:19:27.825 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-22T01:19:27.905 INFO:teuthology.orchestra.run.smithi203.stderr: 92.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-22T01:19:27.964 INFO:teuthology.orchestra.run.smithi111.stderr: 92.5% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-22T01:19:28.404 INFO:teuthology.orchestra.run.smithi111.stderr: 92.8% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T01:19:28.455 INFO:teuthology.orchestra.run.smithi038.stderr: 92.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T01:19:29.361 INFO:teuthology.orchestra.run.smithi111.stderr: 92.7% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T01:19:29.421 INFO:teuthology.orchestra.run.smithi038.stderr: 92.9% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T01:19:29.517 INFO:teuthology.orchestra.run.smithi038.stderr: 92.8% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T01:19:29.705 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T01:19:29.990 INFO:teuthology.orchestra.run.smithi111.stderr: 92.7% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T01:19:29.992 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-22T01:19:29.992 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m5.915s 2024-04-22T01:19:29.992 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m21.325s 2024-04-22T01:19:29.992 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.735s 2024-04-22T01:19:30.377 INFO:teuthology.orchestra.run.smithi038.stderr: 92.7% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-22T01:19:30.379 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-22T01:19:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m6.305s 2024-04-22T01:19:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m21.556s 2024-04-22T01:19:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.718s 2024-04-22T01:19:31.083 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/caf31b2e-0043-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T01:19:31.086 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-22T01:19:31.086 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m7.009s 2024-04-22T01:19:31.086 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m20.076s 2024-04-22T01:19:31.086 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.711s 2024-04-22T01:19:31.086 INFO:tasks.cephadm:Archiving logs... 2024-04-22T01:19:31.086 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi038/log 2024-04-22T01:19:31.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T01:19:34.386 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi111/log 2024-04-22T01:19:34.387 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T01:19:37.572 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi203/log 2024-04-22T01:19:37.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T01:19:40.924 INFO:tasks.cephadm:Removing cluster... 2024-04-22T01:19:40.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --force 2024-04-22T01:19:41.124 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:19:41.415 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --force 2024-04-22T01:19:41.605 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:19:41.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid caf31b2e-0043-11ef-bc93-c7b262605968 --force 2024-04-22T01:19:42.077 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: caf31b2e-0043-11ef-bc93-c7b262605968 2024-04-22T01:19:42.361 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T01:19:42.361 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T01:19:42.380 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T01:19:42.396 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T01:19:42.411 INFO:tasks.cephadm:Teardown complete 2024-04-22T01:19:42.412 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T01:19:42.423 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T01:19:42.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T01:19:42.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T01:19:42.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T01:19:42.550 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T01:19:42.560 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T01:19:42.560 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:19:42.563 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:19:42.566 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:19:42.578 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:19:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:19:42.579 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:19:42.582 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:19:42.583 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-22T01:19:42.583 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +37us[ +37us] +/- 77ms 2024-04-22T01:19:42.583 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 +123us[ +131us] +/- 73ms 2024-04-22T01:19:42.583 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -140us[ -133us] +/- 67ms 2024-04-22T01:19:42.583 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +29us[ +29us] +/- 77ms 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 +123us[ +129us] +/- 73ms 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -148us[ -148us] +/- 67ms 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 27 +78us[ +93us] +/- 78ms 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 7 377 26 +151us[ +167us] +/- 73ms 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -149us[ -134us] +/- 67ms 2024-04-22T01:19:42.585 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:19:42.586 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T01:19:42.597 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T01:19:42.597 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T01:19:42.608 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T01:19:42.709 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T01:19:42.773 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T01:19:42.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:19:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:42.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:19:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:42.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:19:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:42.921 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T01:19:42.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T01:19:42.960 INFO:teuthology.task.internal:Duration was 1514.138005 seconds 2024-04-22T01:19:42.961 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T01:19:42.983 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T01:19:42.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T01:19:42.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T01:19:42.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T01:19:43.024 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:19:43.026 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:19:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:19:43.432 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T01:19:43.433 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T01:19:43.433 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T01:19:43.456 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-04-22T01:19:43.457 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T01:19:43.498 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-22T01:19:43.499 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T01:19:43.526 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T01:19:43.526 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T01:19:43.529 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T01:19:43.541 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T01:19:43.591 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T01:19:43.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T01:19:43.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T01:19:43.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T01:19:44.009 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T01:19:44.020 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T01:19:44.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T01:19:44.045 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T01:19:44.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T01:19:44.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T01:19:44.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:44.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:44.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:44.129 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-22T01:19:44.131 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-04-22T01:19:44.155 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-22T01:19:44.178 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:44.197 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:44.225 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:19:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:19:44.239 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T01:19:44.251 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T01:19:44.252 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi038 2024-04-22T01:19:44.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T01:19:44.294 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi111 2024-04-22T01:19:44.294 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T01:19:44.328 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667718/remote/smithi203 2024-04-22T01:19:44.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T01:19:44.363 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T01:19:44.363 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T01:19:44.365 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T01:19:44.368 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T01:19:44.412 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T01:19:44.426 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T01:19:44.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T01:19:44.438 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T01:19:44.438 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T01:19:44.441 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T01:19:44.443 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T01:19:44.453 INFO:teuthology.orchestra.run.smithi038.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 01:19 /home/ubuntu/cephtest 2024-04-22T01:19:44.455 INFO:teuthology.orchestra.run.smithi111.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 01:19 /home/ubuntu/cephtest 2024-04-22T01:19:44.466 INFO:teuthology.orchestra.run.smithi203.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 01:19 /home/ubuntu/cephtest 2024-04-22T01:19:44.467 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T01:19:44.479 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T01:19:44.506 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} duration: 1514.1380054950714 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-22T01:19:44.506 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T01:19:44.588 INFO:teuthology.run:pass